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Mission 

Our  mission  is  to  provide  independent,  relevant,  and  timely  oversight 
of  the  Department  of  Defense  that  supports  the  warfighter;  promotes 
accountability,  integrity,  and  efficiency;  advises  the  Secretary  of 
Defense  and  Congress;  and  informs  the  public. 


Vision 

Our  vision  is  to  be  a  model  oversight  organization  in  the  Federal 
Government  by  leading  change,  speaking  truth,  and  promoting 
excellence — a  diverse  organization,  working  together  as  one 
professional  team,  recognized  as  leaders  in  our  field. 
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Objective 

Our  objective  for  this  audit  was  to 
evaluate  the  Space  and  Naval  Warfare 
Systems  Command  (SPAWAR]  process  to 
justify,  review,  and  approve  requests  for 
waivers  of  criteria  to  certify  readiness 
for  operational  testing  and  deferrals  of 
operational  testing  requirements.  We  also 
summarized  the  results  of  our  evaluation 
of  the  Navy's  management  of  waivers  and 
deferrals  from  operational  test  requirements 
for  the  nine  Navy  acquisition  programs 
reviewed.  This  report  is  the  third  in  a 
series  of  reports  that  will  evaluate  the 
Navy's  management  of  its  waivers  and 
deferrals  for  acquisition  programs. 

Findings 

The  Navy  Multiband  Terminal  [NMT] 
program  manager  did  not  request  waivers 
when  the  program  did  not  meet  the 
certification  criteria  needed  to  enter  initial 
operational  test  and  evaluation  (IOT&E). 
Additionally,  the  program  managers  for 
the  NMT,  Tactical  Mobile,  Digital  Modular 
Radio,  and  Computer  Network  Defense 
programs  held  operational  test  readiness 
review  (OTRR)  briefings  that  did  not  fully 
document  that  they  had  met  the  certification 
criteria  for  entering  IOT&E. 

These  conditions  occurred  because  Navy 
policy  on  requesting  waivers  was  unclear. 

In  addition,  in  May  2009,  SPAWAR  canceled 
their  policy  on  waivers  and  deferrals. 


As  a  result,  the  NMT  program  completed  IOT&E  with 
deficiencies  that  diminished  the  system's  ability  to  perform 
its  primary  communication  mission.  Additionally,  the 
incomplete  OTRR  briefings  hindered  the  program  executive 
officer  from  fully  considering  the  readiness  of  programs 
for  IOT&E. 

Overall,  Navy  program  managers  and  system  sponsors  did 
not  fully  implement  Navy  policies  for  requesting  waivers 
and  deferrals  before  certifying  program  readiness  for  IOT&E 
to  support  the  final  production  decision.  Of  nine  Navy 
acquisition  programs  that  entered  final  production  between 
April  2012  and  April  2014,  program  managers  on:  four  did 
not  request  waivers  when  not  meeting  all  IOT&E  certification 
requirements;  five  had  OTRR  briefings  not  fully  documenting 
how  they  met  certification  criteria;  and  one  did  not  request 
deferrals  from  testing  that  planned  to  demonstrate  system 
requirements  during  IOT&E.  These  conditions  occurred 
because  Navy  policy  was  unclear  on  when  program  managers 
had  to  request  waivers  and  deferrals.  Additionally,  Navy 
system  sponsors  for  one  program  did  not  obtain  an  agreement 
from  the  Joint  Chiefs  of  Staff  (JCS)  that  the  deferral  would 
not  unacceptably  affect  military  use  before  independently 
granting  the  deferral.  This  occurred  because  Navy  policy 
did  not  require  notifying  JCS  on  deferrals.  As  a  result,  six  of 
nine  programs  reviewed  completed  IOT&E  with  unresolved 
deficiencies  that  negatively  impacted  primary  missions. 

Recommendations 

We  recommend  the  Commander,  Space  and  Naval  Warfare 
Systems  Command,  update  Command  policy  to  include 
implementing  the  planned  revision  of  Secretary  of  the  Navy 
Instruction  5000. 2E,  "Department  of  the  Navy  Implementation 
and  Operation  of  the  Defense  Acquisition  System  and  the 
Joint  Capabilities  Integration  and  Development  System." 

We  are  not  making  additional  recommendations  on  the 
Navy's  management  of  waivers  and  deferrals.  We  made  these 
recommendations  in  two  earlier  reports,  as  discussed  in 
Finding  B,  and  the  Navy  has  ongoing  corrective  actions. 
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Space  and  Naval  Warfare  Systems  Command  and  Overall 
Navy  Needs  to  Improve  Management  of  Waiver  and 
Deferral  Requests 


Management  Comments  and 
Our  Response 

The  Deputy  Department  of  the  Navy  Test  and  Evaluation 
Executive,  responding  for  the  Commander,  Space  and 
Naval  Warfare  Systems  Command,  agreed,  stating  that 
SPAWAR  plans  to  distribute  an  updated  Command  policy 
that  will  clarify  guidance  requiring  SPAWAR  programs 
to  follow  the  OTRR  waiver  process,  as  documented 
in  SECNAVINST  5000. 2E.  The  Deputy  stated  the 
updated  Command  policy  is  expected  to  be  released  by 
November  2015.  The  Deputy  addressed  the  specifics 
of  the  recommendation,  and  no  further  comments  are 
required.  Please  see  the  Recommendation  Table  on  the 
following  page. 
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INSPECTOR  GENERAL 

DEPARTMENT  OF  DEFENSE 

4800  MARK  CENTER  DRIVE 
ALEXANDRIA,  VIRGINIA 22350-1 500 

October  8,  2015 

MEMORANDUM  FOR  UNDER  SECRETARY  OF  DEFENSE  FOR  ACQUISITION,  TECHNOLOGY, 

AND  LOGISTICS 

VICE  CHAIRMAN  OF  THE  JOINT  CHIEFS  OF  STAFF 
DIRECTOR,  OPERATIONAL  TEST  AND  EVALUATION 
NAVAL  INSPECTOR  GENERAL 

SUBJECT:  Space  and  Naval  Warfare  Systems  Command  and  Overall  Navy  Needs  to  Improve 
Management  of  Waiver  and  Deferral  Requests  (Report  No.  DODIG-2016-003) 

We  are  providing  this  report  for  your  information  and  use.  We  determined  that  the  Space  and 
Naval  Warfare  Systems  Command,  as  well  as  Naval  Air  and  Naval  Sea  Systems  Commands, 
should  improve  management  of  waiver  and  deferral  requests  to  allow  the  program  executive 
officer  to  better  determine  program  readiness  for  entering  testing  to  support  the  final 
production  decision.  We  conducted  this  audit  in  accordance  with  generally  accepted 
government  auditing  standards. 

We  considered  management  comments  on  a  draft  of  this  report  when  preparing  the  final 
report.  Comments  from  the  Deputy,  Department  of  the  Navy  Test  and  Evaluation 
Executive,  responding  for  the  Commander,  Space  and  Naval  Warfare  Systems  Command, 
addressed  all  specifics  of  the  recommendation  and  conformed  to  the  requirements  of 
DoD  Instruction  7650.03;  therefore,  we  do  not  require  additional  comments. 

We  appreciate  the  courtesies  extended  to  the  staff.  Please  direct  questions  to  me  at 
(703)  604-9077  (DSN  664-9077). 


■Jacqueline  L.  Wicecarver 
Assistant  Inspector  General 
Acquisition,  Parts,  and  Inventory 
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Introduction 


Objective 

Our  objective  for  this  audit  was  to  evaluate  the  Space  and  Naval  Warfare  Systems 
Command  process  to  justify,  review,  and  approve  requests  for  waivers  of  criteria 
to  certify  readiness  for  operational  testing  and  deferrals  of  operational  testing 
requirements.  We  also  summarized  the  results  of  our  evaluation  of  the  Navy’s 
management  of  waivers  and  deferrals  from  operational  test  requirements  for  the 
nine  Navy  acquisition  programs  reviewed.  This  report  is  the  third  in  a  series  of 
reports  that  will  evaluate  the  Navy’s  management  of  its  waivers  and  deferrals 
for  acquisition  programs.  See  Appendix  A  for  a  discussion  of  our  scope  and 
methodology  and  prior  audit  coverage  related  to  the  audit  objectives. 

Navy  Policy  on  Waivers  and  Deferrals 

Secretary  of  the  Navy  Instruction  5000. 2 E 

Secretary  of  the  Navy  Instruction  (SECNAVINST)  5000. 2E1  establishes  criteria 
to  certify  Navy  systems  as  ready  to  enter  into  initial  operational  test  and 
evaluation  (IOT&E).2  IOT&E  precedes  the  full-rate  production  (final  production) 
decision.  The  instruction  requires  the  Systems  Command  commander,  program 
executive  officer  (PEO),  and  program  manager  to  conduct  an  operational  test 
readiness  review  (OTRR)  to  certify  system  readiness  to  begin  IOT&E. 

An  OTRR  is  a  product  and  process  assessment  to  make  sure  a  system  can  proceed 
into  IOT&E  with  a  high  probability  of  successfully  completing  operational  testing. 
Upon  completing  the  OTRR,  if  the  System  Command  commander,  PEO,  and  program 
manager  determine  the  system  is  ready  to  enter  IOT&E,  they  must  either  certify 
to  the: 

•  Commander,  Operational  Test  and  Evaluation  Force  that  the  system 
is  ready  for  IOT&E,  as  required  by  the  Test  and  Evaluation  Master 
Plan  (TEMP),  with  no  waivers  or  deferrals  requested;  or 

•  Chief  of  Naval  Operations  (CNO)  (N84);  Director,  Innovation,  Test  and 
Evaluation,  and  Technology  Requirements,  that  the  system  is  ready  for 
IOT&E,  with  requests  for  waivers  or  deferrals. 


1  Secretary  of  the  Navy  Instruction  (SECNAVINST)  5000. 2E,  "Department  of  the  Navy  Implementation  and  Operation  of 
the  Defense  Acquisition  System  and  the  Joint  Capabilities  Integration  and  Development  System/'  September  1,  2011. 

2  Initial  Operational  Test  and  Evaluation  is  the  dedicated  operational  test  and  evaluation  conducted  on  production 
representative  articles  to  determine  whether  systems  are  operationally  effective  and  suitable  to  support  a  final 
production  decision. 
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SECNAVINST  5000. 2E  also  states  that  when  waiver  or  deferral  requests 
are  anticipated,  the  program  manager  must  coordinate  with  the  program 
sponsor;  CNO  (N84),  Director,  Innovation,  Test,  and  Evaluation  and  Technology 
Requirements;  and  the  Commander,  Operational  Test  and  Evaluation  Force  before 
the  OTRR.  The  program  sponsor  must  then  formally  concur  with  the  proposed 
waivers  or  deferrals.  Additionally,  when  the  System  Command  commander, 

PEO,  and  program  manager  certify  system  readiness  for  IOT&E  with  waivers 
or  deferrals,  they  must  provide  an  information  copy  of  the  certification  to  the 
program  sponsor  within  the  Office  of  the  CNO.  Concurrence  with  waivers  and 
deferrals  affects  program  execution  as  follows: 

•  Waivers  are  a  deviation  from  the  criteria  identified  for  certifying  IOT&E 
readiness.  Waivers  allow  programs  to  start  IOT&E  without  meeting 

one  or  more  of  the  20  criteria  required  by  SECNAVINST  5000. 2E  to  certify 
readiness  to  enter  IOT&E.  Waivers  do  not  change  or  delay  any  system  or 
testing  requirements. 

•  Deferrals  allow  programs  to  delay  the  testing  of  requirements  identified 
in  the  TEMP,  moving  testing  requirements  from  IOT&E  to  a  later  follow-on 
test  period. 

Waiver  and  deferral  approvals  can  result  in  the  more  rapid  delivery  of  capabilities 
to  operating  Navy  Forces.  However,  the  System  Command  commander,  PEO,  and 
program  manager  must  fully  evaluate  the  potential  impacts  that  waivers  and 
deferrals  have  on  the  mission  capability  delivered.  Waiver  criteria  to  certify 
readiness  for  IOT&E  or  defer  operational  test  requirements  could  result  in 
premature  final  production  decisions.  This  could  create  the  need  for  the  costly 
retrofitting  of  fielded  units  to  deliver  operational  performance  that  is  less  than 
required  to  fully  meet  the  expected  threat.  CNO  (N84)  staff  said  their  "approval" 
of  waivers  and  deferrals  indicates  that  the  program  manager  followed  the  OTRR 
process  necessary  to  certify  program  readiness  for  IOT&E  despite  risks. 

Secretary  of  the  Navy  Manual  M -5000. 2 

Secretary  of  the  Navy  Manual  M-5000.23  provides  additional  discretionary  guidance 
relating  to  the  Navy’s  management  of  waivers  and  deferrals. 


3  Secretary  of  the  Navy  Manual  M-5000.2,  "Acquisition  and  Capabilities  Guidebook,"  May  9,  2012,  Section  4.6, 
"Certification  of  Readiness  for  Operational  Testing." 
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Space  and  Naval  Warfare  Systems  Command 

Space  and  Naval  Warfare  Systems  Command  (SPAWAR)  develops,  delivers,  and 
sustains  communications  and  information  capabilities  for  warfighters,  keeping 
them  connected  anytime,  anywhere.  With  a  space-support  activity,  two  system 
centers,  and  partnerships  with  three  program  executive  offices,  SPAWAR  provides 
the  hardware  and  software  needed  to  execute  Navy  missions. 

Background  on  Programs  Selected  for  Review 

We  identified  four  SPAWAR  acquisition  programs  that  received  final  production 
decisions  from  April  14,  2012,  through  April  14,  2014:  Navy  Multiband 
Terminal  (NMT),  Digital  Modular  Radio  (DMR),  Tactical  Mobile  (TacMobile), 
and  Computer  Network  Defense  (CND).  The  following  sections  describe  these 
programs,  based  on  documentation  the  program  managers  provided.  See 
Appendix  B  for  more  information  on  the  Naval  Air  Systems  Command  (NAVAIR) 
and  Naval  Sea  Systems  Command  (NAVSEA)  programs  we  reviewed  during  our 
evaluation  of  the  Navy’s  overall  management  of  waivers  and  deferrals  from 
operational  test  requirements. 

Navy  Multiband  Terminal 

NMT  is  an  Acquisition  Category  IC4  major  defense  acquisition  program  that  had 
its  final  production  decision  on  November  30,  2012.  NMT  is  the  next-generation 
maritime  military  satellite  communications  terminal.  The  NMT  system  is  required 
for  the  Advanced  Extremely  High  Frequency  (AEHF)  system,  which  enhances  the 
Navy’s  protected  and  survivable  satellite  communications. 

Digital  Modular  Radio 

DMR  is  an  Acquisition  Category  III5  program  that  had  its  final  production  decision 
on  May  7,  2012.  The  DMR  is  a  digital,  modular,  programmable,  multi-channel, 
multi-function  and  multi-band  radio  system.  The  DMR  provides  improvements 
for  fleet  radio  high-frequency,  very-high-frequency,  and  ultra-high-frequency  band 
capabilities.  DMR  replaces  various  legacy  radio  systems  operating  on  Navy  surface 
ships,  submarines,  and  shore  sites. 


4  Acquisition  category  IC  is  a  program  for  which  the  Head  of  the  DoD  Component  estimates  eventual  total  expenditure  for 
research,  development,  test,  and  evaluation  of  more  than  $480  million  in  FY  2014  constant  dollars  or,  for  procurement, 
more  than  $2.79  billion  in  FY  2014  constant  dollars. 

5  Acquisition  category  III  is  an  acquisition  program  for  which  the  DoD  Component  Head  estimates  eventual  total 
expenditure  for  research,  development,  test,  and  evaluation  of  less  than  $185  million  in  FY  2014  constant  dollars  or, 
for  procurement,  less  than  $835  million  in  FY  2014  constant  dollars. 
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Tactical  Mobile 

TacMobile  is  an  Acquisition  Category  III  program  that  had  its  final  production 
decision  on  December  5,  2012.  The  TacMobile  provides  Maritime  Patrol  and 
Reconnaissance  Force  and  Naval  Component  Commanders  with  the  capability  to 
plan,  direct,  and  control  the  tactical  operations  of  joint  and  naval  expeditionary 
forces  and  other  assigned  units  within  their  respective  areas  of  operation.  The 
TacMobile  program  designs,  develops,  fields,  and  sustains  three  major  operational 
systems:  the  stationary  and  mobile  tactical  operations  centers  and  the  Joint  Mobile 
Ashore  Support  Terminal. 

Computer  Network  Defense 

CND  is  an  Acquisition  Category  IVT6  program  that  had  its  final  production  decision 
on  November  28,  2012.  The  CND  program  includes  commercial  hardware  and 
software  components  that  interoperate  with  the  Navy’s  afloat  and  shore  cyber 
architecture.  CND  detects  and  responds  to  unauthorized  activity  with  Navy 
information  systems  and  networks  afloat  and  on  the  shore. 

Review  of  Internal  Controls 

DoD  Instruction  5010.40,  "Managers’  Internal  Control  Program  Procedures," 

May  30,  2013,  requires  DoD  organizations  to  implement  a  comprehensive  system 
of  internal  controls  that  provides  reasonable  assurance  that  programs  are 
operating  as  intended  and  that  evaluates  the  effectiveness  of  the  controls.  We 
identified  an  internal  control  weakness  with  SPAWAR’s  lack  of  implementing 
guidance  for  requesting  waivers  to  obtain  certification  to  enter  IOT&E.  We  will 
provide  a  copy  of  the  report  to  the  senior  official  responsible  for  internal  controls 
in  the  Department  of  the  Navy. 


Only  the  Navy  and  Marine  Corps  use  acquisition  category  IVT  for  programs  that  are  not  otherwise  designated  at 
acquisition  category  III. 
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Finding  A 

Space  and  Naval  Warfare  Systems  Command  Needs  to 
Improve  Management  of  Waiver  and  Deferral  Requests 

The  NMT  program  manager  did  not  fully  implement  Navy  policy  to  request 
waivers  when  the  program  did  not  meet  all  certification  criteria  to  enter  IOT&E. 
Additionally,  the  program  managers  for  the  NMT,  TacMobile,  DMR,  and  CND 
programs  did  not  include  support  in  the  OTRR  briefings  on  what  they  had  done 
to  meet  the  certification  criteria  for  entering  IOT&E. 

These  conditions  occurred  because  Navy  policy  did  not  clearly  state  that  program 
managers  must  request  waivers  whenever  their  program  does  not  meet  all  criteria 
required  to  start  IOT&E.  Additionally,  in  May  2009,  SPAWAR  canceled  their  policy 
on  waivers  and  deferrals. 

As  a  result,  the  NMT  program  entered  and  completed  IOT&E  with  unresolved 
deficiencies  that  affected  system  reliability  and  software  maturity.  These 
deficiencies  diminished  the  NMT’s  ability  to  perform  its  primary  mission  to  provide 
Navy  forces  with  worldwide  secure,  protected,  and  survivable  communications. 
Additionally,  because  the  program  managers  for  the  four  programs  did  not  prepare 
documentation  for  the  OTRR  briefings  that  captured  program  accomplishments 
against  each  certification  requirement,  the  PEO  and  other  stakeholders  could  not 
fully  consider  program  readiness  for  entering  IOT&E  during  the  OTRR  briefing. 


Waivers  Not  Requested  for  Unmet 
Certification  Requirements 

The  NMT  program  manager  did  not  meet  3  of  20  criteria  required  by 
SECNAVINST  5000. 2D7  to  certify  readiness  for  IOT&E.  Although  the  program 
manager  requested  a  waiver  from  the  criterion  for  having  an  approved  TEMP,  he 
did  not  request  a  waiver  for  two  additional  criteria.  Specifically,  the  program 
manager  did  not  request  waivers  when: 

•  the  TEMP  system  performance  requirements  were  not,  or  were  not 
projected  to  be,  satisfied;  and 

•  software  was  not  sufficiently  mature  and  stable  for  fleet  use. 


7  SECNAVINST  5000. 2D,  "Implementation  and  Operation  of  the  Defense  Acquisition  System  and  the  Joint  Capabilities 
Integration  and  Development  System,"  October  16,  2008. 
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System  Reliability  Requirements  Not  Met 

The  NMT  program  manager  did  not  request  waivers  when  the  program  did  not 
meet  TEMP  performance  requirements  for  system  reliability.  The  TEMP8  identified 
the  average  time  between  critical  failures  and  the  average  time  between  failures  as 
a  key  system  attribute  (secondary  requirement).  The  developmental  test  results9 
showed  that  the  NMT  was  not  reliable.  Tests  that  used  the  latest  software  update 
showed  that  the  average  time  between  NMT  failures  rose  from  251.8  hours  to 
338.12  hours.  However,  reliability  was  still  significantly  less  than  the  1,100  hours 
between  failures  specified  in  the  TEMP.  SECNAVINST  5000. 2D  requires  systems 
to  meet  performance  requirements  identified  in  the  TEMP.  Even  before  the  OTRR, 
the  reliability  of  the  NMT  was  a  concern  that  the  Commander,  Operational  Test 
and  Evaluation  Force,  identified  during  the  operational  assessment  to  support  the 
low-rate  initial  production  decision. 


The  Commander,  Operational  Test  and  Evaluation  Force,10  rated  the  reliability  of 
the  NMT  as  unsatisfactory,  based  on  software  and  hardware  failures.  The  report 


The  report 
stated  that 
the  NMT  was  not 
reliable  enough 
*■0  complete  its 
ratio 


stated  that  the  NMT  was  not  reliable  enough  to  complete  its 
operational  missions.  Specifically,  the  report  concluded  that 
low  reliability  adversely  impacted  communication  between 
users  on  required  frequency  bands* 11  and  significantly 
diminished  mission  operations.  When  the  NMT  System 
is  not  working,  NMT-equipped  ships  cannot  access 
military  satellite  communications  systems  to  exchange 
mission-required  information.  The  AEHF  satellite 
uses  the  NMT  to  enhance  Navy  Forces'  protection  and 


satellite  communications.  The  figure  below  shows  that 
NMT  interoperates  with  AEHF,  trucks,  planes,  ships,  and  ground  operations.  On 
October  15,  2012,  almost  a  year  after  the  IOT&E  report,  a  memorandum12  from  the 


Commander,  Operational  Test  and  Evaluation  Force,  stated  that  the  NMT  system 
reliability  had  significantly  improved  from  that  identified  during  IOT&E  and  now 
met  the  reliability  threshold  requirement. 


8  "Navy  Multiband  Terminal  Test  and  Evaluation  Master  Plan  (TEMP)  Revision  B,"  March  10,  2011. 

9  "Navy  Multiband  Terminal  (NMT)  Developmental  Test  (DT)  Integrated  Test  (IT)-Cl  Test  Report  (Part  II),"  July  18,  2011. 

10  "Navy  Multiband  Terminal  Operational  Test  Agency  Evaluation  Report  (OT-C1),"  November  29,  2011. 

11  A  specific  range  of  radio  frequencies  divided  among  ranges  from  very  low  frequencies  to  extremely  high  frequencies. 

12  "Verification  of  Correction  of  Deficiencies  of  the  Navy  Multiband  Terminal  System,"  October  15,  2012. 
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Figure.  Operational  View  of  the  Navy  Multiband  Terminal 


Source:  HYPERS,  Inc. 


Software  Not  Sufficiently  Mature 

The  NMT  program  manager  did  not  request  a  waiver  from  meeting  the  criterion 
that  software  is  sufficiently  mature.  SECNAVINST  5000. 2D  requires  program 
managers  to  demonstrate  that  software  is  sufficiently  mature  and  stable.  In 
the  OTRR  briefing  charts,13  the  program  office  identified  software  maturity  as 
a  medium  risk.  Specifically,  the  risk  states  that  if  the  NMT  software  does  not 
demonstrate  sufficient  maturity  that  supports  the  reliability  key  performance 
parameter  (primary  requirement),  then  final  production  may  be  delayed. 

The  Director,  Operational  Test  and  Evaluation,  stated  in  his  assessment14  that 
software  failures  accounted  for  more  than  half  of  the  total  failures  recorded  during 
the  IOT&E.  The  NMT  trainers  taught  the  operators  that  when  a  "fault"  message 
appears  on  a  terminal,  they  should  first  attempt  clearing  the  message  without 
further  action.  If  the  message  returns,  the  operators  should  begin  troubleshooting 
the  system  and  when  necessary,  the  operators  should  reboot  the  terminal  to 
see  if  that  clears  the  fault.  Rebooting  the  system  typically  took  between  5  and 
10  minutes  to  complete.  From  operator  surveys,  79  percent  (11  of  14)  of  users  said 
a  reboot  is  normally  required  to  restore  the  NMT  to  a  normal  run  state.  Therefore, 

13  "Operational  Test  Readiness  Review  (OTRR)  For  Navy  Multiband  Terminal  (NMT)  OT-C1,"  July  13,  2011. 

14  "Director,  Operational  Test  and  Evaluation  Navy  Multiband  Terminal  (NMT)  Initial  Operational  Test  and  Evaluation 
Report/'  November  2011. 
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...the  Director, 
Operational 
Test  and  Evaluation, 
concluded  the  NMT 
was  not  operationally 
suitable  because  of 
immature  software 
and  hardware. 


the  Director,  Operational  Test  and  Evaluation,  concluded 
the  NMT  was  not  operationally  suitable  because  of 
immature  software  and  hardware.  As  a  result,  the 
program  manager  delayed  the  final  production 
decision  for  one  year  to  take  corrective  action  and 
make  sure  the  NMT  was  sufficiently  reliable  to  meet 
the  communication  needs  for  supporting  the  Navy’s 
long-term  deployments.  In  December  2012,  one  year 
after  IOT&E,  the  Director,  Operational  Test  and  Evaluation, 
reported15  that  the  NMT  was  now  operationally  suitable. 


Operational  Test  Readiness  Review  Briefing  Charts 
Need  to  Address  Certification  Criteria 

OTRR  briefing  charts  for  NMT,  TacMobile,  DMR,  and  CND  did  not  show  support 
that  the  program  manager  met  all  20  criteria  on  the  OTRR  Checklist  Summary. 
The  following  table  shows  how  the  OTRR  briefs  for  the  four  programs  reviewed 
did  not  include  support  for  between  1  and  5  of  the  20  certification  criteria  for 
starting  IOT&E. 


Table  1.  Criteria  Not  Supported  in  OTRR  Briefing  Charts 


SECNAVINST  5000.2E  -  OTRR  Criteria1 

NMT2 

TacMobile 

DMR3 

CND 

1 

The  operational  test  and  evaluation  manning 
of  the  system  is  adequate  to  simulate  normal 
operating  conditions. 

X 

X 

2 

All  ranges,  facilities,  and  resources  required 
to  execute  operational  tests,  including 
instrumentation,  simulators,  targets,  expendables, 
and  funding,  have  been  identified  and  are  available. 

X 

3 

The  system  provided  for  operational  tests 
and  evaluation,  including  software,  is 
production  representative. 

X 

X 

X 

4 

All  software  is  sufficiently  mature  and  stable  for 
fleet  introduction. 

X 

5 

For  software  qualification  tests,  a  statement  of 
functionality  that  describes  the  software  capability 
has  been  provided  to  the  Commander,  Operational 
Test  and  Evaluation  Force,  and  CNO  (N84). 

X 

6 

For  information  technology  (IT)  systems,  the 
system  has  been  assigned  a  mission-assurance 
category  and  confidentiality  level.  System 
certification  accreditation  documents  or  a  platform 

IT  designation  letter,  as  applicable,  have  been 
provided  to  the  Operational  Test  Agency. 

X 

X 

Criteria  not  supported  in  briefing  charts: 

1 

3 

1 

5 

1  The  table  includes  OTRR  criteria  not  supported  in  OTRR  briefing  charts  by  one  or  more  programs  reviewed. 

2  We  compared  NMT  OTRR  briefing  charts  against  SECNAVINST  5000. 2D,  which  was  applicable  at  the  time  of  the  NMT  OTRR. 

3  We  compared  DMR  OTRR  briefing  charts  against  SECNAVINST  5000. 2C,  which  was  applicable  at  the  time  of  the  DMR  OTRR. 


15  "Director,  Operational  Test  and  Evaluation,  FY  2012  Annual  Report,"  December  2012. 
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Without  including  the  information  found  in  those  briefing  charts,  the  OTRR  did 
not  fully  inform  the  chairperson  or  other  stakeholders  that  the  program  should 
have  entered  into  IOT&E  supporting  the  final  production  decision.  However,  the 
program  offices  staff  said  the  program  managers  updated  the  PEOs  on  the  status  of 
meeting  the  20  certification  criteria  in  meetings  leading  to  the  OTRR.  In  addition, 
the  program  manager  provided  supporting  documentation  to  show  that  the 
program  met  the  criteria. 

The  OTRR  is  a  multi-disciplined  product  and  process  assessment  that  makes  sure 
the  system  can  proceed  into  IOT&E  with  a  high  probability  of  success,  and  that 
the  system  is  effective  and  suitable.  The  OTRR  is  complete  when  the  service 
acquisition  executive  evaluates  and  determines  system  readiness  for  IOT&E.  The 
program  manager  should  conduct  the  OTRR  before  certifying  readiness  for  IOT&E. 
IOT&E  helps  support  the  milestone  decision  authority’s  final  production  decision. 

SPAWAR  Canceled  Policy  on  Waivers  and  Deferrals 

The  deficiencies  in  managing  waiver  requests  and  presenting  insufficient  OTRR 
briefings  occurred  because  Navy  policy  did  not  clearly  state  that  program 
managers  must  request  waivers  whenever  their  program  does  not  meet  all  criteria 
required  to  start  IOT&E.  In  May  2009,  SPAWAR  Notice  396016  canceled  the 
SPAWAR  Instruction  3960. 3E, 17  which  included  a  checklist  to  certify  readiness  for 
operational  evaluation.  The  SPAWAR  Notice  3960  does  not  include  any  guidance  on 
the  OTRR  process  or  when  and  how  to  request  waivers  and  deferrals. 

Navy  policy18  requires  periodic  reviews  of  directives  making  sure  they  comply 
with  standards  and  procedures  and  recommends  that  the  issuing  authority 
conduct  those  reviews  annually.  According  to  SPAWAR  staff,  the  SPAWAR 
Test  and  Evaluation  Integrated  Product  Team  is  currently  developing  SPAWAR 
Instruction  3960. 3F.19  The  Commander,  SPAWAR  should  make  sure  the  SPAWAR 
Instruction  3960. 3F,  "Test  and  Evaluation,”  references  and  provides  guidance  to 
implement  the  revised  Secretary  of  the  Navy  Instruction  5000. 2E. 


16  SPAWAR  Notice  3960,  "SPAWAR  5.0  Guidance,  Standard  Policies,  Procedures  and  Process  for  Test,  Evaluation, 
and  Certification  (TE&C)  for  Command,  Control,  Communications,  Computers,  Intelligence,  Surveillance,  and 
Reconnaissance  (C4ISR)  Systems,  Space  Systems  (SS),  and  Enterprise  Information  Systems  (EIS),"  May  6,  2009. 

17  SPAWAR  Instruction  3960. 3E,  "Test  and  Evaluation,"  June  23,  1992. 

18  OPNAVINST  5215.17,  "Navy  Directives  Issuance  System,"  June  13,  2005,  Enclosure  1,  paragraph  3b  (4). 

19  SPAWAR  Instruction  3960. 3F,  "Test  and  Evaluation,"  expected  to  be  issued  in  late  2015. 
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Conclusion 

The  NMT  entered  and  completed  IOT&E  with  unresolved  deficiencies  that  affected 
software  maturity  and  reliability.  NMT  did  not  fully  meet  its  primary  requirements 
to  provide  Navy  Forces  with  worldwide  secure,  protected,  and  survivable  AEHF 
satellite  communications.  The  program  manager  delayed  the  final  production 
decision  for  one  year  to  take  corrective  action  and  make  sure  the  NMT  was 
sufficiently  reliable  to  meet  the  communication  needs  for  supporting  the  Navy's 
long-term  deployments. 

Additionally,  because  the  program  managers  for  the  four  programs  did  not  prepare 
documentation  for  the  OTRR  briefings  that  captured  program  accomplishments 
against  each  certification  requirement,  the  PEO  and  other  stakeholders  could  not 
fully  consider  program  readiness  for  entering  IOT&E  during  the  OTRR  briefing. 

Department  of  the  Navy  Comments  on  the  Finding 
and  Our  Response 

The  Deputy  Department  of  the  Navy  Test  and  Evaluation  Executive,  responding 
for  the  Commander,  Space  and  Naval  Warfare  Systems  Command  provided  the 
following  comments  on  the  finding.  The  Deputy  also  provided  comments  on  the 
Inspector  General,  DoD  process  for  staffing  the  discussion  draft.  Appendix  D 
provides  a  summary  of  the  comments  on  the  discussion  draft  and  our  response. 

For  the  full  text  of  the  Deputy’s  comments,  see  the  Management  Comments  section 
of  the  report. 

The  Deputy  partially  agreed  with  the  finding  and  acknowledged  that  the  policy  in 
SECNAVINST  5000. 2E  needs  updating  to  clarify  leadership  expectations  of  OTRR 
processes.  The  Deputy  also  stated  that  an  update  to  SECNAV  5000. 2E  was  ongoing. 
The  following  pages  summarize  the  Deputy’s  disagreements  on  the  adequacy  of 
the  Navy’s  processes  for  requesting  waivers  and  conducting  OTRRs,  the  Navy’s 
application  of  the  OTRR  process  for  the  NMT  program,  and  the  relevancy  of  the 
SPAWAR  policy  the  finding  references. 

Adequacy  of  the  Navy's  Processes  for  Requesting  Waivers  and  Conducting 
Operational  Test  Readiness  Reviews 

The  Deputy  stated  the  report’s  finding  -  because  program  managers  did  not  request 
waivers,  the  systems  did  not  meet  performance  requirements,  and  the  PEO  could 
not  effectively  consider  program  readiness  for  IOT&E  -  incorrectly  implies  that 
the  only  information  discussed  at  the  OTRR  is  about  certification  criteria  not  met. 
Additionally,  the  finding  incorrectly  implies  the  only  information  provided  to  the 
PEO  to  support  his  or  her  decision  is  during  the  OTRR. 
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The  Deputy  stated  the  OTRR  is  a  culmination  of  many  months  of  program  office 
and  stakeholder  effort  to  determine  the  status  of  the  20  certification  criteria  and 
how  much  detail  to  include  in  the  OTRR  briefing  charts.  In  addition,  there  are 
many  layers  of  expert  reviews  making  sure  that  the  PEO  has  objective  data  to 
make  an  informed  decision  at  the  OTRR. 

The  Deputy  also  stated  the  finding  was  incomplete  because  it  did  not  state  that 
the  four  program  managers  provided  documentation  before  the  OTRR  briefings 
that  captured  program  accomplishments  against  each  certification  criteria.  It 
is  efficient  to  encourage  the  program  manager  to  provide  and  successfully 
coordinate  this  documentation  through  official  forums,  like  the  test  and  evaluation 
working  integrated  product  team,  which  includes  staff  from  the  PEO,  program 
office,  and  other  stakeholders.  Individuals  on  the  test  and  evaluation  working 
integrated  product  team  can  inform  their  leadership  before  the  OTRR  on  the 
status  of  documentation  that  captured  program  accomplishments  against  each 
certification  criteria. 

Our  Response 

We  recognize  during  OTRR  program  staff  provide  the  PEO  and  other  stakeholders 
with  information  beyond  how  the  program  manager  accomplished  criteria  to 
certify  readiness  for  IOT&E  and  that  there  are  multiple  levels  of  program  review. 
However,  when  program  managers  request  waivers,  it  serves  to  highlight  to  the 
PEO  and  other  stakeholders  specific  shortfalls  from  the  criteria  that  programs 
must  normally  meet,  according  to  SECNAVINST  5000. 2E,  to  demonstrate  they  are 
ready  to  enter  IOT&E.  The  Navy  has  long  recognized  the  importance  of  meeting 
these  IOT&E  certification  criteria  and  of  requesting  waivers  for  any  deviations. 
Specifically,  with  only  minor  wording  changes,  the  same  20  certification 
criteria  and  provisions  to  request  waivers  in  the  SECNAVINST  5000. 2E  were 
also  provided  in  earlier  versions  of  this  Instruction,  back  to  and  including 
SECNAVINST  5000. 2C,  dated  November  19,  2004.  In  addition,  the  Deputy  agreed, 
to  issue  interim  guidance  to  make  sure  the  OTRR  briefing  charts  include  a 
summary  that  discusses  each  of  the  20  certification  criterions  in  adequate  detail 
to  support  the  program  office  assessment  in  his  response  to  a  recommendation  in 
Report  No.  DODIG-2015-122.20  This  guidance  will  go  into  effect  before  the  Navy 
completes  its  overall  update  of  the  SECNAVINST  5000. 2E. 

We  disagree  with  the  Deputy’s  statement  that  the  finding  was  incomplete  because 
it  did  not  state  that  the  four  program  managers  provided  documentation  before  the 
OTRR  briefings  that  captured  program  accomplishments  against  each  certification 
criteria.  The  report  did  state  that  the  four  programs  provided  documentation 

20  Report  No.  DODIG-2015-122,  "Naval  Air  Systems  Command  Needs  to  Improve  Management  of  Waiver  Requests," 

May  15,  2015. 
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before  the  OTRR  showing  that  the  program  met  the  20  certification  criteria.  In 
the  finding  section  "Operational  Test  Readiness  Review  Briefing  Charts  Need  to 
Address  Certification  Criteria"  we  acknowledge  that  program  office  staff  stated  the 
program  managers  updated  the  PEOs  on  the  status  of  meeting  the  20  certification 
criteria  in  meetings  before  the  OTRR.  Program  office  staff  also  stated  that  the 
program  manager  provided  supporting  documentation  to  show  the  program  met 
the  criteria. 

Operational  Test  Readiness  Review  Briefing  Charts  Need  to  Address 
Certification  Criteria 

The  Deputy  stated  that  the  finding  statements  on  the  NMT  program  are: 

•  inaccurate  because  they  conclude  in  Table  1  that  the  program  did  not 
meet  a  certification  criterion  relating  to  manning  levels.  Specifically, 
the  program  office  reported  the  manning  levels  to  simulate  normal 
operating  conditions  in  the  NMT  OTRR  briefing  charts.  Further,  the  NMT 
Government  Test  Team  requested  the  test  assets  for  the  IOT&E  via  the 
fleet  service  request  process.  This  information  was  included  in  the  NMT 
Development  Test/Integrated  Test  Cl  site  briefs  and  the  2010  TEMP. 

•  incomplete  because  they  do  not  state  that  the  program  manager  delayed 
the  final  production  decision  following  IOT&E  to  successfully  correct 
and  verify  the  deficiencies  identified  during  IOT&E.  The  Commander, 
Operational  Test  and  Evaluation  Force,  concluded  the  NMT  was 
operationally  suitable  in  October  2012  and  the  Director,  Operational 
Test  and  Evaluation,  reported  the  NMT  was  operationally  suitable  in 
December  2012. 

•  inaccurate  and  incomplete  because  the  approved  NMT  2011  TEMP 
documented  that  testing,  prior  to  the  availability  of  the  AEHF  satellite, 
could  evaluate  the  NMT’s  compatibility,  functions,  and  whether  it  could 
perform  its  operational  missions.  The  NMT  program  manager  was 
directed  to  document  the  plan  to  test  the  extended  data  rate  capability 
during  follow-on  operational  testing,  in  the  2010  and  2011  TEMP.  The 
Director,  Operational  Test  and  Evaluation,  approved  the  IOT&E  plan  with 
full  knowledge  of  this  limitation. 

Our  Response 

We  disagree  with  the  Deputy’s  comments  that  Table  1  is  inaccurate  regarding  NMT 
program  not  meeting  the  certification  criterion  relating  to  manning  levels.  We 
did  not  state  that  the  NMT  program  did  not  meet  the  manning  levels  certification 
criteria.  The  table  in  the  report  identifies  OTRR  criteria  not  supported  in  OTRR 
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briefing  charts  by  one  or  more  programs  reviewed.  The  NMT  OTRR  briefing  charts 
listed  the  manning  level  certification  criteria  as  met;  however,  there  were  no 
details  provided  on  how  the  program  met  the  criteria  or  a  reference  to  the  TEMP 
where  manning  was  reported.  Including  this  information  in  the  OTRR  briefing 
charts  would  better  inform  the  chairperson  or  other  stakeholders  responsible  for 
determining  whether  the  program  should  enter  into  IOT&E  supporting  the  final 
production  decision. 

We  agree  the  report  did  not  state  that  the  NMT  program  manager  delayed  the  final 
production  decision  to  successfully  correct  and  verify  the  deficiencies  identified 
during  IOT&E.  We  revised  the  report  conclusion  section  to  give  the  program 
manager  credit  for  delaying  the  final  production  decision  and  taking  corrective 
action  to  address  the  deficiencies  identified  during  IOT&E. 

We  agree  that  the  NMT  program  manager  had  approval  to  test  the  NMT’s 
capabilities  without  the  AEHF  satellite  being  available  and  deleted  this  section 
in  our  draft  report. 

Relevancy  of  the  Space  and  Naval  Warfare  Systems  Command  Policy  the 
Finding  References 

The  Deputy  stated  the  finding  was  not  current  or  accurate  because  the  referenced 
"internal  guidance"  was  an  unsigned  draft  SPAWAR  policy  that  had  not  been 
distributed  to  the  program.  Therefore,  the  comparisons  made  in  Appendix  C  of 
the  draft  report  are  not  relevant,  as  they  compare  an  unsigned,  undistributed 
policy  memorandum  to  the  SECNAVINST  5000.2  criteria.  In  addition,  the  Deputy 
stated  that  the  comparison  was  not  relevant  because  the  approved  PEO  operational 
guide  was  distributed  before  any  of  the  PEO  Command,  Control,  Communications, 
Computers  and  Intelligence  and  Space  programs  were  ready  for  IOT&E. 

Our  Response 

Based  on  the  Deputy  comments  that  the  "internal  guidance"  referenced  in  the 
draft  report  was  a  draft  policy  that  was  not  signed  or  distributed  to  the  program 
office,  we  deleted  references  to  the  "internal  guidance,”  including  the  comparisons 
of  the  SECNAV  and  PEO  policies  that  showed  the  six  criteria  the  PEO  policy  did 
not  include. 
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Recommendation  and  Our  Response 

We  recommend  the  Commander,  Space  and  Naval  Warfare  Systems  Command,  to 
make  sure  the  Space  and  Naval  Warfare  Instruction  3960.3F,  "Test  and  Evaluation," 
references  and  provides  guidance  to  implement  the  Navy  policy  in  the  planned 
revision  of  Secretary  of  the  Navy  Instruction  5000.2E,  "Department  of  the  Navy 
Implementation  and  Operation  of  the  Defense  Acquisition  System  and  the  Joint 
Capabilities  Integration  and  Development  System.” 

Department  of  the  Navy 

The  Deputy  Department  of  the  Navy  Test  and  Evaluation  Executive,  responding 
for  the  Commander,  Space  and  Naval  Warfare  Systems  Command,  agreed, 
stating  that  SPAWAR  plans  to  distribute  an  updated  Test  and  Evaluation  policy 
(SPAWAR  Instruction  3960.5).  He  stated  the  updated  instruction  will  address 
roles  and  responsibilities  and  will  clarify  guidance  requiring  SPAWAR  programs 
to  follow  the  OTRR  waiver  process,  as  documented  in  SECNAVINST  5000. 2E.  The 
Deputy  stated  the  updated  SPAWAR  Instruction  3960.5  is  expected  to  be  released 
by  November  2015. 

Our  Response 

The  Deputy  addressed  the  specifics  of  the  recommendation,  and  no  further 
comments  are  required. 
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Finding  B 

Navy  Systems  Commands  Need  to  Improve 
Management  of  Waivers  and  Deferrals 

Navy  program  managers  and  system  sponsors  within  the  Office  of  the  CNO  did 
not  fully  implement  Navy  policies  for  requesting  waivers  and  deferrals  before 
certifying  program  readiness  for  IOT&E  supporting  the  final  production  decision. 
Our  review  of  nine21  Navy  acquisition  programs  that  entered  final  production 
between  April  14,  2012,  and  April  14,  2014,  showed  that  program  managers  on: 

•  four  programs  did  not  request  waivers  when  those  programs  did  not  meet 
all  IOT&E  certification  requirements; 

•  five  programs  had  OTRR  briefings  that  did  not  clearly  document  what 
they  had  done  to  meet  certification  criteria;  and 

•  one  program  did  not  request  deferrals  from  testing  planned  to 
demonstrate  system  requirements  during  IOT&E. 

These  conditions  occurred  because  Navy  policy  did  not  clearly  state  that  program 
managers  must  request  a  waiver  whenever  a  program  does  not  meet  all  criteria 
required  to  enter  IOT&E  and  request  a  deferral  to  delay  testing. 

Additionally,  Navy  system  sponsors  for  one  program  did  not  obtain  agreement 
from  the  Joint  Chiefs  of  Staff  (JCS)  that  three  deferrals  would  not  unacceptably 
affect  military  use  before  independently  granting  the  deferrals.  This  occurred 
because  Navy  policy  did  not  require  sponsors  to  communicate  to  the  JCS  their 
decision  to  grant  a  deferral  to  a  program  designated  as  Acquisition  Category  I, 
or  of  JCS  interest. 

As  a  result,  program  manager  and  sponsor  implementation  of  Navy  waiver  and 
deferral  policy  permitted  six  of  nine  programs  reviewed  to  complete  IOT&E  with 
unresolved  deficiencies  that  negatively  impacted  performing  the  primary  missions. 


21  We  found  that  one  of  nine  programs  reviewed  had  multiple  problems  implementing  Navy  policy. 


DODIG-2016-003  |  15 


Finding  B 


Program  Should  Have  Requested  Additional  Waivers 
or  Deferrals 


For  four  of  nine  programs  reviewed,  we  found  that  program  managers  did  not 
request  waivers  when  those  programs  did  not  meet  all  IOT&E  certification 
requirements.  The  following  sections  describe  the  criteria  that  were  not  met 
and  how  these  shortfalls  diminished  mission  effectiveness. 

P-8A  Poseidon  Aircraft  Program 

DODIG-2015-122,22  reported  the  P-8A  program  manager  did  not  meet  4  of 
20  criteria  specified  in  SECNAVINST  5000. 2E  to  certify  readiness  for  IOT&E. 
Although  the  P-8A  program  manager  requested  a  waiver  from  the  flight-deficiency 
criteria,  he  did  not  request  a  waiver  for  three  additional  criteria.  Specifically,  the 
program  manager  did  not  request  waivers  when: 

•  the  TEMP  system  performance  requirements  were  not,  or  were  not 
projected  to  be,  satisfied; 

•  necessary  logistics  support  (spares,  repair  parts,  and  equipment)  was  not 
available  for  IOT&E;  and 

•  the  Joint  Interoperability  Test  Command  had  not  concurred  that  program 
interoperability23  was  sufficient  to  enter  IOT&E. 


As  a  result,  the  P-8A  entered  and  completed  IOT&E  with 


unresolved  deficiencies  that  affected  threat  detection, 


maneuverability,  and  information  exchange.  These 


deficiencies  diminished  the  P-8A’s  ability  to  perform 
the  aircraft’s  primary  missions  of  anti-submarine 
warfare;  anti-surface  warfare;  and  intelligence, 
surveillance,  and  reconnaissance.  During  follow-on 
testing,  the  P-8A  did  not  meet  performance 


thresholds  for  four  critical  technical  parameters 
and  two  effectiveness  measures. 


Distributed  Targeting  System 

DODIG-2015-122  identified  that  the  Distributed  Targeting  System  (DTS)  program 
manager  did  not  request  a  waiver  when  performance  requirements  in  the  TEMP 
for  system  reliability  were  not,  or  were  not  projected  to  be,  satisfied.  As  a 
result,  the  DTS  entered  and  completed  IOT&E  with  unresolved  deficiencies  that 
affected  reliability.  The  unresolved  deficiencies  reduced  the  DTS's  reliability 
to  perform  the  amphibious  warfare  mission.  Almost  8  months  after  the  IOT&E 


22  Report  No.  DODIG-2015-122,  "Naval  Air  Systems  Command  Needs  to  Improve  Management  of  Waiver  Requests," 
May  15,  2015. 

23  The  ability  to  support  military  operations  and  effectively  exchange  information. 
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report,  the  memorandum24  from  the  Commander,  Operational  Test  and  Evaluation 
Force,  did  not  include  the  reliability  deficiency  as  one  that  the  program  manager 
had  corrected. 

Identity  Dominance  System 

DODIG-2015-17225  identified  that  the  Identity  Dominance  System  (IDS)  program 
manager  did  not  request  a  waiver  for  system  performance  requirements  identified 
in  the  TEMP  to  match  fingerprints  in  a  timely  manner  against 
those  in  a  100,000-person  watch  list.  In  addition,  the  ID 
program  manager  did  not  comply  with  the  requirement 
in  SECNAVINST  5000. 2E  to  request  deferrals  through 
the  CNO  (N84)  for  delays  in  testing  requirements 
directed  in  the  TEMP.  The  delayed  testing  involved 
demonstrating  23  system  characteristics  from  the 
program  requirements  document.  As  a  result,  the  IDS 
program  completed  IOT&E  with  unresolved  deficiencies 
and  reduced  mission  effectiveness.  The  IDS  deficiencies 
slowed  the  system’s  ability  to  match  fingerprints  against  terrorists 
and  other  persons  of  interest  in  watch  lists.  At  final  production,  the  program 
manager  had  accepted  32  IDS  systems  valued  at  $1.1  million  that  could  not  fully 
perform  the  assigned  missions. 

Navy  Multiband  Terminal 

As  discussed  in  Finding  A  of  this  report,  the  NMT  program  manager  did  not  meet 
3  of  20  criteria  SECNAVINST  5000. 2D  requires  to  certify  for  IOT&E  readiness. 
Although  the  program  manager  requested  a  waiver  from  the  criterion  for  having 
an  approved  TEMP,  he  did  not  request  a  waiver  for  two  additional  criteria. 
Specifically,  the  program  manager  did  not  request  waivers  when: 

•  the  TEMP  system  performance  requirements  were  not,  or  were  not 
projected  to  be,  satisfied;  and 

•  software  was  not  sufficiently  mature  and  stable  for  fleet  use. 

As  a  result,  the  NMT  entered  and  completed  IOT&E  with  unresolved  deficiencies 
that  affected  system  reliability  and  software  maturity.  NMT  did  not  fully  meet  its 
primary  requirements  to  provide  Navy  Forces  with  worldwide  secure,  protected, 
and  survivable  communications.  On  October  15,  2012,  almost  one  year  after  the 
IOT&E  report,  the  Commander,  Operational  Test  and  Evaluation  Force,26  said  the 
NMT  system  reliability  had  significantly  improved  from  that  identified  during 
IOT&E  and  now  met  the  reliability  requirement. 


The  IDS 
deficiencies 
slowed  the  system’s 
ability  to  match 
fingerprints  against 
terrorists  and  other 
persons  of  interest 
in  watch  lists. 


24  "Verification  of  Correction  of  Deficiencies  (VDC)  of  the  Distributed  Targeting  System  (DTS),"  August  1,  2013. 

25  Report  No.  DODIG-2015-172,  "Naval  Sea  Systems  Command  Needs  to  Improve  Management  of  Waiver  and  Deferrals 
Requests,"  September  14,  2015. 

26  "Verification  of  Correction  of  Deficiencies  of  the  Navy  Multiband  Terminal  System,"  October  15,  2012. 
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Lack  of  Evidence  of  Accomplishments  in  Briefing  Charts 
for  Operational  Test  Readiness  Reviews 

For  five  of  nine  programs  reviewed,  we  found  OTRR  briefings  did  not  clearly 
document  what  they  had  done  to  meet  certification  criteria.  DODIG-2015-122 
stated  the  OTRR  briefing  charts  for  the  E-2D  from  February  1,  2012,  including 
backup  charts,  did  not  show  evidence  of  how  the  program  manager  met  9  of 
20  certification  criteria  on  the  OTRR  Checklist  Summary.  However,  after  reviewing 
briefing  charts,27  which  were  not  part  of  the  OTRR  briefing,  we  found  support  for 
those  nine  criteria.  Without  including  the  information  found  in  those  briefing 
charts,  the  OTRR  did  not  fully  inform  the  chairperson  or  other  stakeholders  as 
to  whether  the  program  should  have  entered  into  IOT&E  in  support  of  the  final 
production  decision.  Because  of  our  audit,  the  Navy  is  revising  its  policy  that 
clarifies  the  need  to  include  a  summary  in  the  OTRR  briefing  charts  and  make  sure 
the  summary  discusses  each  criterion  in  adequate  detail  to  support  the  program 
office  assessment.  See  Appendix  C  for  details  on  past  recommendations  and 
Navy-planned  corrective  actions. 

As  discussed  in  Finding  A  of  this  report,  the  OTRR  briefing  charts  for  NMT, 
TacMobile,  DMR,  and  CND  did  not  show  evidence  that  the  program  manager  met 
all  20  criteria  on  the  OTRR  Checklist  Summary.  Table  1,  in  Finding  A,  lists  the 
six  criteria  not  supported  in  OTRR  briefing  charts  by  one  or  more  programs 
reviewed.  Without  including  the  information  found  in  those  briefing  charts,  the 
OTRR  did  not  fully  inform  the  chairperson  or  other  stakeholders  that  the  program 
should  have  entered  into  IOT&E  supporting  the  final  production  decision. 

Programs  Should  Have  Had  Joint  Chiefs  of  Staff 
Assessment  of  Deferrals 

For  one  of  nine  programs  reviewed,  Navy  system  sponsors  did  not  obtain  JCS 
agreement  that  three  deferrals  would  not  affect  military  use  before  independently 
granting  the  deferral.  DODIG-2015-172  showed  that  the  Director,  Surface 
Warfare  (N96),  Standard  Missile-6,  (SM-6)  did  not  communicate  with  the  JCS 
concerning  these  deferrals  until  nearly  two  years  after  delaying  the  testing  of 
primary  requirements.  The  Joint  Chiefs  of  Staff  Instruction28  requires  sponsors 
and  the  JCS  interaction  when  there  are  schedule  changes  that  require  validating 


27  "E-2D  AHE  Road  to  IOT&E  Deep  Dive/  January  12,  2012. 

28  Chairman,  Joint  Chiefs  of  Staff  Instruction  3170.011,  "Joint  Capabilities  Integration  and  Development  System," 
January  23,  2015,  Enclosure  A,  "Joint  Capabilities  Integration  and  Development  System,"  page  A-17  Section  (f) 
"Event  Driven  Reviews." 
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primary  requirement  modifications.  This  interaction  allows  JCS  to  assess  continued 
military  system  usefulness  and  may  result  in  a  program  evaluation  or  modification 
to  production  increments.29 

The  report  also  states  the  JCS’s  assessment  of  the  impact  of  approved  deferrals  was 
delayed  almost  two  years  after  the  sponsor  granted  them.  This  occurred  because 
the  SM-6  program  sponsor  did  not  notify  the  JCS  in  a  timel 
manner  about  deferring  testing  of  the  system’s  ability  to 
meet  three  primary  requirements.  The  SM-6  deferrals 
involved  delaying  testing  of  missile  range,  availability, 
and  interoperability.  At  the  time  of  the  final  production 
decision,  SM-6  program  staff  stated  that  the  Navy  had 
accepted  41  SM-6  missiles  valued  at  $148.3  million.  While 
CNO  staff  stated  the  systems  improved  existing  capability, 
they  could  not  fully  perform  their  assigned  missions. 

Waivers  or  Deferrals  Diminished  Mission  Effectiveness 

For  six  of  nine  programs  reviewed,  the  waivers  or  deferrals  resulted  in  diminished 
mission  effectiveness  of  the  systems  delivered.  DODIG-2015-122  reported  the 
P-8A,  E-2D,  and  the  DTS  programs  entered  and  completed  IOT&E  with  unresolved 
deficiencies  that  affected  threat  detection,  maneuverability,  information  exchange, 
and  reliability.  DODIG-2015-172  identified  that  the  IDS  completed  IOT&E  with 
unresolved  deficiencies  that  affected  its  ability  to  detect  terrorists.  Additionally, 
the  report  identified  that  the  SM-6  completed  IOT&E  without  demonstrating 
primary  system  requirements  for  increased  missile  range,  availability,  and 
interoperability.  As  discussed  in  Finding  A  of  this  report,  the  NMT  entered  and 
completed  IOT&E  with  unresolved  deficiencies  that  affected  software  maturity 
and  reliability. 

Summary 

We  found  that: 

•  four  of  nine  programs  should  have  requested  additional  waiver 
or  deferrals; 

•  five  of  nine  programs  lacked  evidence  of  accomplishments  in  OTRR 
briefing  charts; 

•  one  of  nine  programs  should  have  had  JCS  assessments  of  deferrals;  and 

•  six  of  nine  programs  had  waivers  or  deferrals  that  diminished 
mission  effectiveness. 


The  SM-6 
deferrals 
involved  delaying 
testing  of  missile 
range,  launch 
availability,  and 
interoperability. 


29  "Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and  Development  System/'  (JCIDS),  February  12,  2015 
version  -  Appendix  A  to  Enclosure  D,  "Development  of  Key  Performance  Parameters,  Key  System  Attributes,  and 
Additional;  Performance  Attributes,  Section  1,  "Overview;"  January  19,  2012  version  -  Appendix  A  to  Enclosure  B, 
"Key  Performance  Parameters  and  Key  System  Attributes,"  Section  1,  "Overview;"  and  January  31,  2011  version  - 
Enclosure  B,  "Performance  Attributes  and  Key  Performance  Parameters,"  paragraph  l.b. 


DODIG-2016-003  |  19 


Finding  B 


The  following  table  summarizes  results  for  each  program  reviewed. 


Table  2.  Summary  of  Audit  Results  by  Command  and  Program 


System 

Commands 

Programs 

Reviewed 

Should  Have 
Requested 
Additional 
Waiver  or 
Deferrals 

Lack  of 
Evidence  of 
Accomplishments 
in  OTRR  Briefing 
Charts 

Should  Have 
Had  JCS's 
Assessment 
of  Deferrals 

Waivers  or 
Deferrals 
Diminished 
Mission 
Effectiveness 

NAVAIR 

P-8A 

X1 

X 

E-2D 

X 

X 

DTS 

X 

X 

NAVSEA 

SM-6 

X 

IDS 

X1'2 

X 

SPAWAR 

NMT 

X1 

X 

X 

TacMobile 

X 

DMR 

X 

CND 

X 

1  The  program  manager  did  not  request  waivers  for  all  certification  criteria  not  met. 

2  The  program  manager  did  not  request  deferrals  for  IOT&E  testing  no  longer  planned. 


Planned  Management  Action 

The  Deputy  Department  of  the  Navy  Test  and  Evaluation  Executive  plans  to  issue 
interim  guidance  to  address  the  waiver  and  deferral  process,  which  will  go  into 
effect  before  the  Navy  completes  its  overall  update  of  the  SECNAVINST  5000. 2E. 
The  interim  guidance  will: 

•  replace  the  term  "waiver"  with  "deviation  from  SECNAV  policy," 
meaning  the  PEO  is  notifying  the  Assistant  Secretary  of  the  Navy  for 
Research,  Development,  and  Acquisition  that  a  program  will  proceed  to 
operational  testing  without  achieving  one  or  more  of  the  20  specified 
certification  criteria; 

•  replace  the  term  "deferral"  with  "deferral  of  test  requirements,"  which 
will  apply  to  a  delay  in  testing  capabilities  identified  in  the  current 
requirements  document  and  agreed  to  in  the  TEMP; 

•  emphasize  that  the  PEO  identify  which  of  the  20  certification  criteria 
did  not  meet  requests  for  a  deviation  in  the  OTRR  certification  message 
and  state  why  the  decision  was  made  to  proceed  to  operational  testing 
without  having  met  those  specific  criteria;  and 

•  clarify  the  need  to  include  a  summary  of  the  program  office  assessment 
of  the  20  certification  criteria  in  the  OTRR  briefing  charts  and  make  sure 
the  summary  discusses  each  criterion  in  adequate  detail  to  support  the 
program  office  assessment. 

Therefore,  we  did  not  make  additional  recommendations. 
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Management  Comments  on  the  Finding  and 
Our  Response 

Although  not  required  to  comment,  the  Deputy  Department  of  the  Navy  Test 
and  Evaluation  Executive  provided  comments  on  the  finding.  A  summary  of  the 
Deputy’s  comments,  along  with  our  response,  follows  below.  For  the  full  text  of 
the  Deputy’s  comments,  see  the  Management  Comments  section  of  the  report. 

Department  of  the  Navy  Comments 

The  Deputy  partially  agreed  with  the  finding,  and  provided  the  following 
comments  on  the  portions  of  the  finding  where  he  did  not  agree. 

•  The  finding  implies  the  only  way  the  program  manager  and  PEOs  are 
informed  of  a  system’s  required  capabilities  and  determines  whether 
the  system  is  ready  to  proceed  to  operational  testing  is  through 
reading  a  detailed  requirements  document.  The  Deputy  stated  that 
this  incorrectly  assumed  the  requirement  document  is  the  only  vehicle 
for  conveying  programmatic  information  to  the  PEO  and  program 
manager.  Rather,  resource  sponsors,  program  managers,  and  PEOs 
are  in  daily  communications  discussing  risk,  cost,  performance,  and 
schedule;  numerous  technical  and  fiscal  realities,  as  well  as  actual  system 
capabilities,  all  affect  the  decision  to  test  or  field  the  system  with  a 
specific  performance  level. 

•  Adding  the  requirement  for  the  Joint  Requirements  Oversight  Council 
of  the  JCS  to  approve  waivers  or  deferrals  would  be  a  significant 
administrative  burden.  SECNAVINST  5000. 2E,  paragraph  4. 6. 2. 2,  states 
that  when  PEOs  certify  readiness  for  operational  tests  with  exceptions, 
they  must  send  a  message  to  CNO  N84  and  provide  copies  to  the  program 
sponsor  for  their  formal  concurrence,  and  to  the  Assistant  Secretary  of 
the  Navy  for  Research,  Development,  and  Acquisition;  and  the  Commander, 
Operational  Test  and  Evaluation  Force.  PEOs  have  Title  10  responsibility, 
delegated  from  the  CNO,  to  develop  requirements  for  systems  that  will 
equip  the  fleet.  The  SECNAVINST  5002. E  policy  ensures  that  the  PEOs 
have  executed  that  responsibility  and  have  determined  system  capabilities 
are  adequate  before  proceeding  to  operational  testing.  The  Navy  views 
the  existing  SECNAV  policy  as  providing  adequate  oversight. 
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Our  Response 

We  recognize  that  during  the  OTRR,  program  staff  provided  the  PEO  and  other 
stakeholders  with  information  beyond  how  the  programs  accomplished  the  criteria 
to  certify  readiness  for  IOT&E,  and  that  there  are  multiple  levels  of  program 
review.  However,  when  waivers  are  requested  by  program  managers,  they  serve 
to  highlight  to  the  PEO  and  other  stakeholders  the  specific  shortfalls  from  the 
criteria  that  programs  must  normally  meet,  according  to  SECNAVINST  5000. 2E,  to 
demonstrate  that  they  are  ready  to  enter  IOT&E. 

Regarding  the  need  for  JCS  approval  of  waivers  and  deferrals  in  DODIG-2015-172,30 
we  revised  that  report  to  state  that  the  PEO  and  CNO  staff,  and  not  the  JCS,  were 
responsible  for  assessing  whether  system  shortfalls  impact  meeting  program 
requirements  when  certifying  a  system  to  proceed  to  IOT&E.  However,  the  JCS 
Deputy  Chief  commented  that  a  "potential  gap  in  the  risk-assessment  mechanism" 
existed  whenever  service  managers  decided  internally  to  defer  operational  testing 
of  primary  system  requirements.  These  deferrals  can  enable  programs  like  the 
SM-6  to  move  forward  to  meet  initial  operational  capability  dates  and  to  avoid 
the  requirement  for  a  JCS  Joint  Requirements  Oversight  Council  review.  The 
Deputy  Chief  suggested  we  add  a  recommendation  to  revise  JCIDS  procedures  to 
close  this  risk-assessment  gap.  As  a  result,  we  added  a  recommendation  in  the 
NAVSEA  final  report  to  the  Vice  Chairman,  JCS  to  revise  the  JCIDS  procedures  to 
close  the  risk  assessment  gap.  We  also  revised  our  original  recommendation,  in 
Report  No.  DODIG-2015-172,  to  the  Secretary  of  the  Navy  to  recommend  that  he 
revise  the  SECNAVINST  5000. 2E,  to  align  with  the  revisions  the  Vice  Chairman, 
Joint  Chiefs  of  Staff,  makes  to  the  JCIDS  Manual. 


30  Report  No.  DODIG-2015-172,  "Naval  Sea  Systems  Command  Needs  to  Improve  Management  of  Waiver  and  Deferrals 
Requests/  September  14,  2015. 
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Appendix  A 

Scope  and  Methodology 

We  conducted  this  performance  audit  from  March  2015  through  July  2015  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives.  We  believe  the  evidence  obtained  provides  a 
reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit  objectives. 

We  interviewed  key  personnel  and  performed  fieldwork  at  the 
following  organizations: 

•  Joint  Staff  J-8  Force  Structure,  Resource,  and  Assessment,  Washington  D.C.; 

•  Director,  Operational  Test  and  Evaluation,  Washington  D.C.; 

•  Assistant  Secretary  of  the  Navy  for  Research,  Development, 
and  Acquisition,  Washington  D.C.; 

•  Director,  Innovation,  Test,  and  Evaluation,  and  Technology  Requirements, 
Washington  D.C.; 

•  Commander,  Operational  Test  and  Evaluation  Force,  Norfolk,  Virginia; 

•  P-8A  Poseidon  Aircraft  Program  Office,  Patuxent,  Maryland; 

•  E-2D  Advanced  Hawkeye  Aircraft  Program  Office,  Patuxent,  Maryland; 

•  Distributed  Targeting  System  Program  Office,  Patuxent,  Maryland; 

•  Identity  Dominance  System  Program  Office,  Washington  D.C.; 

•  Standard  Missile-6  Program  Office,  Arlington,  Virginia; 

•  Navy  Multiband  Terminal  Program  Office,  San  Diego,  California; 

•  Digital  Modular  Radio  Program  Office,  San  Diego,  California; 

•  Tactical  Mobile  Program  Office,  San  Diego,  California;  and 

•  Computer  Network  Defense  Program  Office,  San  Diego,  California. 

We  collected,  reviewed,  and  analyzed  documents  dated  April  1988  through 
February  2015.  We  reviewed  requirements  documents,  test  and  evaluation  plans 
and  reports,  operational  test  readiness  review  checklists  and  briefing  charts,  and 
program  certification  messages  that  requested  waivers  and  deferrals  to  determine 
whether  Navy  staff  adequately  justified,  reviewed,  and  approved  the  waiver  and 
deferral  of  operational  testing  requirements  for  the  acquisition  of  Navy  systems. 
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Additionally,  we  reviewed  program  planning  and  reporting  documents 
and  compared  them  to  the  policies  and  guidance  in  the  following  DoD 
and  Navy  issuances. 

•  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  3170. 01H,  "Joint 
Capabilities  Integration  and  Development  System,”  January  10,  2012; 

•  "Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and 
Development  System,"  January  19,  2012; 

•  "Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and 
Development  System,"  February  12,  2015; 

•  Defense  Acquisition  Guidebook,  September  16,  2013; 

•  Secretary  of  the  Navy  Instruction  5000. 2E,  "Department  of  the  Navy 
Implementation  and  Operation  of  the  Defense  Acquisition  System 
and  the  Joint  Capabilities  Integration  and  Development  System," 
September  1,  2011; 

•  Secretary  of  the  Navy  Manual  5000.2,  "Department  of  the  Navy 
Acquisition  and  Capabilities  Guidebook,"  May  9,  2012; 

•  NAVAIR  Instruction  3960. 2D,  "Acquisition  Test  and  Evaluation," 

May  30,  2012;  and 

•  NAVSEA  Instruction  3960. 2D  "Test  and  Evaluation,"  April  22,  1988. 

Selection  of  Programs  to  Review 

We  obtained  a  query  from  the  database  of  the  Assistant  Secretary  of  the  Navy 
for  Research,  Development  and  Acquisition  Information  Systems  to  identify  Navy 
systems-acquisition  programs  that  received  a  final  production  decision  from 
April  14,  2012,  through  April  14,  2014.  We  identified  nine  Navy  programs  and 
conducted  a  100  percent  review  of  those  programs  to  evaluate  the  command’s 
management  of  the  waiver  and  deferral  process. 

Use  of  Computer-Processed  Data 

We  did  not  use  computer-processed  data  to  perform  this  audit. 

Prior  Coverage 

During  the  last  5  years,  the  Department  of  Defense  Inspector  General  (DoD  IG) 
issued  three  reports  that  discussed  operational  test  waivers  and  deferrals. 
Unrestricted  DoD  IG  reports  can  be  accessed  at 
http://www.dodig.mil/pubs/index.cfm. 
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DoD  IG 

Report  No.  DODIG-2015-172,  "Naval  Sea  Systems  Command  Needs  to  Improve 
Management  of  Waiver  Requests,”  September  14,  2015 

Report  No.  DODIG-2015-122,  "Naval  Air  Systems  Command  Needs  to  Improve 
Management  of  Waiver  Requests,”  May  15,  2015 

Report  No.  DODIG-2013-088,  "The  Navy  P-8A  Poseidon  Aircraft  Needs  Additional 
Critical  Testing  Before  the  Full-Rate  Production  Decision,"  June  10,  2013 
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Appendix  B 

Background  on  Programs  Selected  for  Review 

We  identified  nine  acquisition  programs  that  received  final  production  decisions 
from  April  14,  2012,  through  April  14,  2014:  the  P-8A  Poseidon  aircraft, 

E-2D  Advanced  Hawkeye  aircraft,  Distributed  Targeting  System  (DTS), 

Identity  Dominance  System  (IDS),  Standard  Missile-6  (SM-6),  Navy  Multiband 
Terminal  (NMT),  Digital  Modular  Radio  (DMR),  Tactical  Mobile  (TacMobile)  and 
Computer  Network  Defense  (CND).  The  following  sections  describe  the  NAVAIR 
and  NAVSEA  programs  we  reviewed.  We  discuss  the  SPAWAR  programs  in  the 
Introduction  section  of  this  report. 

P-8A  Poseidon  Aircraft 

The  P-8A  Poseidon  aircraft  program  is  an  Acquisition  Category  ID31  major 
defense-acquisition  program  that  had  its  final  production  decision  on 
January  3,  2014.  The  Navy  designed  the  P-8A  Poseidon  to  replace  the  aging 
P-3C  Orion  aircraft.  Like  the  P-3C,  the  P-8A  aircraft  provides  capabilities  for 
three  principal  missions: 

•  anti-submarine  warfare; 

•  anti-surface  warfare;  and 

•  intelligence,  surveillance,  and  reconnaissance  in  maritime  operations. 

The  Capability  Production  Document32  (the  system  requirements  document)  states 
that  the  anti-submarine  warfare  mission,  which  detects,  tracks,  and  destroys  or 
neutralizes  hostile  submarines,  was  the  primary  reason  the  Navy  invested  in  the 
P-8A  aircraft.  The  anti-surface  warfare  mission  provides  maritime  superiority 
against  surface  vessels  and  a  common  sea-surface  picture.  The  intelligence, 
surveillance,  and  reconnaissance  mission  provides  a  flexible  and  responsive 
intelligence-gathering  capability  in  support  of  Joint,  Naval,  and  national  interests. 

The  Navy  sought  to  expedite  procuring  and  deploying  the  P-8A  Poseidon  because  of 
concerns  with  airframe  corrosion  on  the  P-3C  Orion.  While  P-3C  Orion  fatigue  has 
remained  a  persistent  risk,  the  Navy  has  inspection,  repair,  and  modification  efforts 
in  place  to  sustain  the  P-3C  Orion  fleet  until  the  P-8A  Poseidon  began  replacing  the 
P-3C  Orion  in  2013.  As  discussed  in  DoD  IG  Report  2013-088, 33  the  P-8A  Poseidon 


31  Acquisition  category  ID  is  a  program  for  which  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
estimates  eventual  total  expenditure  for  research,  development,  test,  and  evaluation  of  more  than  $480  million  in 

FY  2014  constant  dollars  or,  for  procurement,  of  more  than  $2.79  billion  in  FY  2014  constant  dollars. 

32  "Capability  Production  Document  for  the  United  States  Navy  P-8A  Poseidon  Multi-Mission  Maritime  Aircraft  (MMA)," 
Increment  1,  June  22,  2009. 

33  Report  No.  DoDIG-2013-088,  "The  Navy  P-8A  Poseidon  Aircraft  Needs  Additional  Critical  Testing  Before  the  Full-Rate 
Production  Decision,"  June  10,  2013. 
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had  uncorrected  system  deficiencies  that  impacted  the  above-mission  capabilities 
entering  IOT&E.  However,  Navy  managers  accepted  the  risk  from  diminished 
capabilities  and  allowed  the  program  to  enter  into  IOT&E,  without  having  to  fully 
correct  the  deficiencies  until  after  the  final  production  decision. 

E-2D  Advanced  Hawkeye  Aircraft 

The  E-2D  Advanced  Hawkeye  aircraft  program  is  an  Acquisition  Category  IC 
major  defense-acquisition  program  that  had  its  final  production  decision  on 
March  1,  2013.  The  E-2D  aircraft  is  an  all-weather,  twin-engine,  carrier-based, 
airborne  command,  control,  and  surveillance  aircraft  designed  to  extend  task 
force  defense  boundaries.  The  aircraft’s  mission  includes: 

•  advance  warning  of  approaching  enemy  surface  units  and  aircraft; 

•  guide  interceptor  or  attack  aircraft; 

•  area  surveillance,  intercept,  and  search  and  rescue; 

•  communications  relay;  and 

•  air  traffic  control. 

The  E-2D  aircraft  replaces  the  E-2C,  multi-mission  airborne  early-warning  and 
airborne  battle-management  command-and-control  aircraft. 

Distributed  Targeting  System 

The  DTS  is  an  Acquisition  Category  III34  program  that  had  its  final  production 
decision  on  April  19,  2013.  The  DTS  will  improve  attack  accuracy  using  on-board 
processing  of  the  F/A-18E/F  Super  Hornet  aircraft.  The  DTS  compares  the 
geographic  imagery  of  the  ground  below  from  aircraft  sensors  against  reference 
images  in  a  database  at  the  National  Geospatial-Intelligence  Agency.  DTS 
generates  a  match  based  on  similarities  between  the  images  to  generate  targeting 
coordinates.  This  capability  will  better  enable  the  F/A-18E/F  aircraft  to  perform 
air-controller,  attack-coordination,  and  reconnaissance  missions  and  to  conduct 
attacks  on  stationary  land  targets,  such  as  mobile  missile  units. 


34  Acquisition  category  III  is  an  acquisition  program  for  which  the  DoD  Component  Head  estimates  eventual  total 
expenditure  for  research,  development,  test,  and  evaluation  of  less  than  $185  million  in  FY  2014  constant  dollars 
or,  for  procurement,  less  than  $835  million  in  FY  2014  constant  dollars. 
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Standard  Missile-6 

SM-6  program  is  an  Acquisition  Category  ID  major  defense  acquisition  program 
that  had  its  final  production  decision  on  July  15,  2013.  The  SM-6  extended-range 
active  missile  will  provide  ship  self-defense,  fleet-area  defense,  and  theater  air 
defense  for  sea-  and  shore-based  forces.  The  SM-6  extended-range  active  missile  is 
a  surface-to-air  supersonic  missile,  launched  from  AEGIS  Cruisers  and  Destroyers, 
capable  of  successfully  engaging  manned  and  unmanned,  fixed-  or  rotary-wing 
aircraft,  and  land-attack  or  anti-ship  cruise  missiles  in  flight. 

Identity  Dominance  System 

The  IDS  is  an  Acquisition  Category  IVT35  program  that  had  its  final 
production  decision  in  June  2013.  The  IDS  is  a  man-portable  biometrics 
collection-and-matching  and  digital-media-collection  system  intended  to  support 
expanded  maritime  interception  operations.  The  fundamental  purpose  of  the  IDS 
is  to  provide  the  Navy  with  an  integrated  capability  to  fix  the  identity  of  unknown 
persons.  There  are  two  key  aspects  of  this  capability.  First,  IDS  will  enable  the 
Navy  to  rapidly  establish  or  verify  the  identity  of  individuals,  and  will  improve  the 
capability  to  update,  manage,  and  share  that  information.  Second,  IDS  will  enable 
the  collection  of  digital  forensic  data  related  to  identity  dominance.  The  system 
will  be  a  combat  multiplier  in  overseas  contingency  operations  where  forces  need 
to  fix  the  identity  of  unidentified  individuals. 


35  Acquisition  programs  in  the  Navy  and  Marine  Corps  not  otherwise  designated  as  ACAT  III  are  designated  ACAT  IV. 
An  ACAT  IVT  program  requires  operational  test  and  evaluation. 
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Appendix  C 

Prior  Recommendations  and  Status 

DODIG-2015-122,  "Naval  Air  Systems  Command  Needs  to  Improve  Management  of 
Waiver  Requests,"  May  15,  2015,  recommended  policy  changes  to: 

•  better  direct  Navy  program  managers  and  weapons  system  sponsors  in 
requesting  waivers  and  deferrals  when  required; 

•  conduct  OTRR  briefings  that  clearly  demonstrate  program 
accomplishments  against  IOT&E  certification  requirements;  and 

•  allow  the  JCS  to  fully  assess  the  impact  of  waivers  and  deferrals  on  the 
military  usefulness  of  weapons  developed  under  programs  designated  as 
Acquisition  Category  I,  or  of  interest  to  the  JCS. 

In  response  to  our  recommendations,  the  Deputy  Department  of  the  Navy  Test 
and  Evaluation  Executive,  responding  for  the  Assistant  Secretary  of  the  Navy  for 
Research,  Development  and  Acquisition,  agreed  to  update  policy  for  requesting 
waivers  and  deferrals,  clarifying  that  those  Navy  managers  must  clearly 
demonstrate  program  accomplishments  in  the  OTTR  briefings.  The  Deputy  also 
plans  to  issue  interim  guidance  to  address  the  waiver  and  deferral  process.  The 
interim  guidance  will: 

•  replace  the  term  "waiver"  with  "deviation  from  SECNAV  policy,"  which 
means  the  PEO  will  notify  the  Assistant  Secretary  of  the  Navy  for 
Research,  Development  and  Acquisition,  that  a  program  is  proceeding  to 
IOT&E  without  achieving  one  or  more  of  the  20  specified  certification 
criteria  for  starting  this  test  phase; 

•  replace  the  term  "deferral"  with  "deferral  of  test  requirements,"  which 
will  apply  to  a  delay  in  testing  capabilities  identified  in  the  current 
requirements  document  and  agreed  to  in  the  TEMP; 

•  emphasize  that,  when  requesting  a  deviation  in  the  OTRR  certification 
message,  the  PEO  identifies  which  of  the  20  certification  criteria  were  not 
met  and  states  why  the  decision  was  made  to  proceed  to  operational  test 
without  meeting  those  specific  criteria;  and 

•  clarify  the  need  to  include  a  summary  in  the  OTRR  briefing  charts  of  the 
program  office  assessment  of  the  20  criteria  and  make  sure  the  summary 
discusses  each  criterion  in  adequate  detail  to  support  the  program 
office  assessment. 


DODIG-2016-003  |  29 


Appendixes 


The  Deputy,  responding  for  the  Assistant  Secretary  of  the  Navy  for  Research, 
Development  and  Acquisition,  did  not  agree  with  a  recommendation  to  require 
sponsors  of  acquisition  programs  designated  as  Acquisition  Category  I,  or  of 
interest  to  the  Joint  Capability  Board  or  the  Joint  Requirements  Oversight  Council, 
to  certify  to  the  Vice  Chairman,  Joint  Chiefs  of  Staff  that: 

•  approved  waivers  of  certification  criteria  do  not  adversely  impact  primary 
requirements;  or 

•  if  a  proposed  waiver  of  certification  criteria,  such  as  the  criteria  to  meet 
system  performance  requirements  in  the  test  and  evaluation  master  plan, 
does  adversely  impact  primary  requirements,  that  the  system  continues 
to  have  military  utility  as  described  in  the  JCIDS  Manual  before  approving 
the  waiver. 

The  Deputy  said  that  adding  a  requirement  to  have  the  JROC  approve  a  waiver 
would  drive  many  flag-level  briefings  before  the  waiver  and  would  even  leave  the 
Navy  to  start  joint  staffing. 

In  response  to  the  Deputy’s  comments,  we  said  the  updated  JCIDS  Manual36  clearly 
states  that  the  validation  authority  is  within  the  Joint  Chiefs  of  Staff  for  Acquisition 
Category  I  programs,  such  as  the  P-8A  and  E-2D  aircrafts,  and  for  programs 
designated  of  interest  to  the  Joint  Capability  Board  or  the  Joint  Requirements 
Oversight  Council  of  the  JCS.  Therefore,  we  requested  that  the  Deputy  provide 
additional  comments  on  the  final  report  that  explain  how  he  would  comply 
with  the  JCIDS  Manual,  but  not  inform  JCS  when  the  PEO  programs  do  not  meet 
primary  requirements. 

DODIG-2015-172,  "Naval  Sea  Systems  Command  Needs  to  Improve  Management  of 
Waiver  and  Deferral  Requests,"  September  14,  2015  recommends  policy  changes  to 
require  sponsors  of  acquisition  programs  designated  as  Acquisition  Category  I  or  of 
interest  to  the  Joint  Chiefs  of  Staff,  to  notify  the  Joint  Chiefs  of  Staff  and  allow  them 
to  re-evaluate  the  program  requirements  before  they  agree  with  deferrals  that 
delay  testing  of  primary  requirements.  The  Deputy  Department  of  the  Navy  Test 
and  Evaluation  Executive,  responding  for  the  Secretary  of  the  Navy,  stated  in  his 
comments  to  the  draft  report  that  the  PEO  and  CNO  staffs,  and  not  the  JCS,  were 
responsible  for  assessing  whether  a  system’s  shortfalls  impact  meeting  program 
requirements  when  certifying  a  system  to  proceed  to  IOT&E.  Therefore,  we  added 
a  recommendation  to  the  Vice  Chairman,  JCS  to  revise  the  JCIDS  procedures  to  close 
the  risk  assessment  gap  and  revised  the  recommendation  to  Secretary  of  the  Navy 
to  recommend  that  he  revise  the  SECNAVINST  5000. 2E,  to  align  with  the  revisions 
the  Vice  Chairman  makes  to  the  JCIDS  Manual.  We  requested  the  Vice  Chairman, 
JCS  and  Secretary  of  the  Navy  to  respond  to  the  final  report. 


36  "Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and  Development  System,"  February  12,  2015 
(updated  JCIDS  Manual). 
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Additionally,  in  DODIG-2015-172,  we  recommended  that  the  Commander,  NAVSEA, 
update  internal  policy  on  waivers  and  deferrals  to  reference  and  implement  Navy 
policy  in  the  planned  revision  of  Secretary  of  the  Navy  Instruction  5000. 2E, 
"Department  of  the  Navy  Implementation  and  Operation  of  the  Defense  Acquisition 
System  and  the  Joint  Capabilities  Integration  and  Development  System."  The 
Deputy  Department  of  the  Navy  Test  and  Evaluation  Executive  responding  for  the 
Commander,  NAVSEA,  agreed  with  the  recommendation. 
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Appendix  D 

Management  Comments  and  Our  Response 

The  Deputy  Department  of  the  Navy  Test  and  Evaluation  Executive,  responding 
for  the  Commander,  Space  and  Naval  Warfare  Systems  Command,  provided 
the  following  comments.  For  the  full  text  of  the  Deputy’s  comments,  see  the 
Management  Comments  section  of  the  report. 

Department  of  the  Navy  Comments 

The  Deputy  stated  that  the  3-day  staffing  time  for  the  discussion  draft  is  not 
adequate  to  allow  for  full  review  and  staffing,  especially  when  delivered  without 
any  warning.  The  Deputy  recommends  that  either  the  audit  team: 

•  eliminate  the  initial  discussion  draft  review  and  only  conduct  the  formal 
30-day  review,  with  a  formal  adjudication  process;  or 

•  extend  the  timeline  for  the  discussion  draft  review  to  7  days  to  provide 
the  Navy  adequate  time  to  thoroughly  review  and  comment  on  the 
discussion  draft. 

The  Deputy  further  stated  that  the  Navy  is  concerned  that  the  audit  team  did 
not  incorporate  previously  provided  comments  to  the  discussion  draft  in  the 
draft  report.  Also,  the  Navy  was  not  provided  an  opportunity  to  adjudicate  in 
order  to  make  sure  the  audit  team  understood  the  Navy  comments.  The  Deputy 
recommends  that  the  audit  team  implement  a  final  comment  adjudication  review 
as  standard  procedure  to  allow  organizations  to  understand  how  the  audit  team 
considered  Navy  comments. 

Our  Response 

The  Deputy’s  comments  that  the  3-day  staffing  time  for  a  discussion  draft  is  not 
adequate  time  for  a  full  review  and  staffing  especially  when  the  discussion  draft  is 
delivered  without  any  warning,  is  misleading.  A  discussion  draft  is  issued  to  senior 
officials  or  their  designated  representatives  to  whom  potential  recommendations 
will  be  directed  and  is  not  meant  to  be  staffed  through  the  chain  of  command.  The 
purpose  of  the  discussion  draft  is  to  make  sure  that  the  report  is  accurate  and  to 
resolve  or  minimize  disagreements  on  conclusions,  findings,  recommendations, 
and  potential  monetary  benefits.  During  the  course  of  the  audit,  we  communicated 
with  senior  SPAWAR  staff  on  what  we  were  finding.  Specifically,  on  April  16,  2015, 
the  audit  team  conducted  tele-conferences  with  the  Deputy  PEO  for  Acquisition 
Management  and  staff  representatives  from  each  SPAWAR  program  reviewed  to 
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discuss  the  issues  that  we  were  finding  with  their  programs.  We  also  provided 
them  with  the  preliminary  copy  of  our  discussion  draft  that  outlined  our  finding 
and  recommendation.  As  a  result  of  these  meetings  we  made  changes  to  the  report. 

Additionally,  we  continued  to  work  with  SPAWAR  staff  to  address  their  comments 
on  the  formal  discussion  draft  report.  Specifically,  we  revised  the  report  relating 
to  the  necessary  resources  for  the  NMT  program  not  being  available  for  operational 
testing.  In  addition,  we  revised  the  report  based  on  SPAWAR  staff  comments  and 
documentation  they  provided  in  response  to  the  discussion  draft  that  showed  the 
program  manager  had  approval  for  delaying  the  NMT  testing. 
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Management  Comments 

Department  of  the  Navy 


DEPARTMENT  OF  THE  NAVY 

OFFICE  OF  THE  CHIEF  OF  NAVAL  OPERATIONS 
2000  NAVY  PENTAGON 
WASHINGTON,  D.C.  20350-2000 


13  Aug  2015 

MEMORANDUM  FOR  INPSECTOR  GENERAL,  DEPARTMENT  OF  DEFENSE 

SUBJECT:  “NAVAL  SYSTEMS  COMMANDS  NEED  TO  IMPROVE  MANAGEMENT  OF 
WAIVER  AND  DEFERRAL  REQUESTS”,  (D2015-D000AE-0157.000) 

The  Navy  has  reviewed  the  subject  report  and  partially  concurs  with  the  findings 
and  recommendations.  SPAWARSYSCOM  (PEO  C4I  and  PMW- 170/A)  and  OPNAV 
(N84/N803)  assisted  the  DODIG  during  the  conduct  of  this  audit,  and  the  report  and 
responses  were  socialized  within  the  offices  of  the  Assistant  Secretary  of  the  Navy 
(Research,  Development  and  Acquisition),  the  Chief  of  Naval  Operations  and 
SPAWARSYSCOM.  Our  response  to  your  draft  report  is  enclosed. 

Responsibility  for  implementing  the  proposed  changes  that  the  Navy  concurs  with 


Carroll  P.  Quade 

Deputy  Department  of  the  Navy 

Test  and  Evaluation  Executive  (N84C) 
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Department  of  the  Navy  (cont'd) 


NAVY  RESPONSE  TO 

DODIG  DRAFT  AUDIT  REPORT  “SPACE  AND  NAVAL  WARFARE  SYSTEMS 
COMMAND  AND  OVERALL  NAVY  NEEDS  TO  IMPROVE  MANAGEMENT 
OF  WAIVER  DEFERRAL  REQUESTS” 

D2015-D000AE-0 157.000,  DATED  2  JULY  2015 

RESULTS  IN  BRIEF  AND  FINDING  A:  Space  and  Naval  Warfare  Systems  Command 
Needs  to  Improve  Management  of  Waiver  and  Deferral  Requests 

The  Navy  Multiband  Terminal  (NMT)  program  manager  did  not  fully  implement  Navy  policy  to 
request  waivers  when  the  program  did  not  meet  all  certification  criteria  to  enter  initial 
operational  test  and  evaluation  (IOT&E).  Additionally,  the  program  managers  for  the  NMT, 
Tactical  Mobile,  Digital  Modular  Radio,  and  Computer  Network  Defense  programs  did  not 
include  support  in  the  operational  test  readiness  review  (OTRR)  briefings  on  what  they  had  done 
to  meet  certification  criteria  for  entering  IOT&E. 

These  conditions  occurred  because  Navy  policy  did  not  clearly  state  that  program  managers  must 
request  waivers  whenever  their  program  does  not  meet  all  criteria  required  to  start  IOT&E.  In 
addition,  in  May  2009,  SPAWAR  canceled  their  policy  on  waivers  and  deferrals. 

As  a  result,  the  NMT  program  entered  and  completed  IOT&E  with  unresolved  deficiencies  that 
affected  system  reliability  and  software  maturity.  These  deficiencies  diminished  the  NMT’s 
ability  to  perform  its  primary  mission  providing  Navy  forces  worldwide  secure,  protected,  and 
survivable  communications.  Additionally,  because  the  program  managers  for  the  four  programs 
did  not  prepare  documentation  for  the  OTRR  briefings  that  captured  program  accomplishments 
against  each  certification  requirement,  the  program  executive  officer  and  other  stakeholders 
could  not  fully  consider  program  readiness  for  entering  IOT&E  during  the  OTRR  briefings. 

N84:  Partially  concur 

This  finding  states  that  because  PMs  did  not  request  waivers,  the  system  did  not  meet 
performance  requirements,  and  the  PEO  could  not  effectively  consider  program  readiness  for 
entering  the  IOT&E  test  phase.  This  implies  that  the  only  information  discussed  at  the 
Operational  Test  Readiness  Review  (OTRR)  is  regarding  certification  criteria  not  met,  and  that 
the  only  information  provided  to  the  PEO  to  support  his  or  her  decision  is  during  the  OTRR. 

This  is  incorrect. 

The  OTRR  is  a  culmination  of  many  months  of  effort  by  the  program  office.  The  program  Test 
and  Evaluation  Working  Integrated  Product  Team  (T&E  WIPT),  which  includes  resource 
sponsors,  DOT&E  and  DASD  (DT&E),  N842,  COTF,  logistics,  technical  test  experts,  training 
and  other  Subject  Matter  Experts  (SMEs),  determines  the  status  of  each  of  the  20  items  of  the 
OTRR  certification  criteria.  The  WIPT  also  determines  the  level  of  detail  that  each  of  the  criteria 
should  be  discussed  at  the  OTRR.  Multiple  iterations  of  the  brief  are  developed  and  reviewed 
before  presenting  to  the  PM  and  the  Deputy  PEO  before  proceeding  to  the  actual  OTRR.  There 
are  many  layers  of  expert  review  to  ensure  that  PEO  is  provided  objective  data  to  make  an 
informed  decision  at  the  OTRR. 

The  existing  SECNAVINST  5000.2  para  4.6.2  states  “The  SYSCOM  commander,  PEO,  and 
DRPM  shall,  unless  otherwise  directed  by  ASN(RD&A)  for  programs  on  the  OSD  T&E  oversight 
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list,  make  one  of  the  following  certifications”.  The  next  two  paragraphs  of  the  instruction  detail  the 
two  certifications,  and  state  that  the  PEO  shall  certify,  by  naval  message,  that  a  system  is  either 
certified  to  proceed  to  Operational  Testing  (OT)  with  no  exceptions,  or  certified  to  proceed  to  OT 
with  T&E  exceptions. 

However,  N84  acknowledges  that  the  current  SECNAV  5000. 2E  policy  could  be  updated  to 
provide  more  clear  guidance  on  leadership  expectations  of  OTRR  processes,  and  a  proposed 
change  is  in  final  staffing. 

SPAWAR  Response: 

PEO  C4I/PMW  170/A:  Do  not  concur. 

The  finding  on  NMT  is  incomplete.  The  DoD  IG  did  not  state  the  NMT  program  manager 
postponed  the  NMT  Full  Rate  Production  (FRP)  decision  following  IOT&E,  and  successfully 
completed  a  Verification  of  Correction  of  Deficiencies  in  August  2012.  The  Commander, 
Operational  Test  and  Evaluation  Force  (COMOPTEVFOR)  concluded  the  NMT  was 
operationally  suitable  on  15  October  2012.  The  Director,  Operational  Test  and  Evaluation 
reported  the  NMT  operationally  suitable  in  December  2012. 

The  second  sentence,  paragraph  1  and  the  last  sentence,  paragraph  3  of  the  Finding  are 
incomplete.  DoD  IG  did  not  state  the  four  program  managers  provided  documentation  before 
the  OTRR  briefings  that  captured  program  accomplishments  against  each  certification 
requirement.  The  Navy  provided  the  DoD  IG  with  documentation  that  captured  program 
accomplishments  against  each  certification  requirement. 

Further,  the  four  program  managers  implemented  the  following  Instructions  and  Better  Buying 
Power  (BBP)  3.0  tenet: 

•  DoD  Instruction  5000.02,  January  15,  2015,  Section  1,  “Purpose”,  paragraph  b  that 
authorizes  tailoring  “the  regulatory  requirements  and  acquisition  procedures  in  this 
instruction  to  more  efficiently  achieve  program  objectives. . .” 

•  SECNAVINST  5000.2E,  1  September  2011,  Section  1.5,  paragraph  1.5.2  (i.e.,  tailoring), 
and  Section  4.6,  paragraphs  4.6.1  (i.e.,  coordinating  tailoring  with  the  T&E  Working 
Integrated  Product  Team  (WIPT)  and  Operational  Test  Authority),  and  4.6.4  (i.e., 
requesting  and  coordinating  waivers  and  deferrals  prior  to  the  OTRR ,  and  using  the 
T&E  WIPT  to  ensure  full  understanding  of  the  impact  on  operational  testing). 

•  BBP  3.0  tenet  Eliminate  Unproductive  Processes  and  Bureaucracy:  Streamline 
documentation  requirements  and  staff  reviews. 

It  is  efficient  to  encourage  program  managers  to  provide  and  successfully  coordinate  this 
documentation  through  official  forums  like  the  T&E  WIPT.  Program  Executive  Officer, 
stakeholders’,  and  program  office  staffs  are  on  the  T&E  WIPT  and  inform  their  respective 
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leadership  before  the  OTRR  on  status  of  documentation  that  captured  program  accomplishments 
against  each  certification  requirement.  Program  managers  can  therefore  better  inform  the 
program  executive  officer  and  stakeholders  at  OTRR  briefings  on  any  risks  and  issues  to 
program  readiness  to  enter  IOT&E. 


L  “Operational  Test  Readiness  Review  Briefing  Charts  Need  to  Address  Certification 

Criteria’9  (Table  and  words) 

OPNAV  Response: 

N84:  Do  not  concur 

As  stated  above,  the  OTRR  brief  is  the  final  action  of  an  extensive  and  robust  decision  process. 
Many  OTRR’s  are  focused  on  the  items  that  require  PEO  awareness  and  attention,  and  where  a 
decision  is  required.  The  prebriefmg  process,  overseen  by  a  SES,  addresses  all  certification 
issues  and  ensures  that  the  OTRR  agenda  is  focused  on  the  most  pertinent  issues. 

SPAWAR  Response: 

PEO  C4I/PMW  170/A:  Do  not  concur. 

The  finding  is  not  accurate.  The  PEO  C4I  does  not  concur  with  the  DoD  IG  concluding  in  Table 
1  that  the  NMT  program  did  not  meet  Criterion  1  titled  “The  operational  test  and  evaluation 
manning  of  the  system  is  adequate  to  simulate  normal  operating  conditions.” 

The  program  office  reported  “The  OT&E  manning  of  the  system  is  adequate  in  numbers,  rates, 
ratings,  and  experience  level  to  simulate  normal  operating  conditions.”  in  the  13  July  2011  NMT 
OTRR  briefing  (refer  to  slide  62  titled  “OTRR  Checklist”  Item  #8).  The  PEO  C4I  provided 
these  slides  to  the  DoD  IG. 

Further,  the  NMT  Government  Test  Team  (GTT)  requested  the  test  assets  for  the  IOT&E  via  the 
Fleet  Service  Request  (FSR)  process.  In  the  FSR,  with  concurrence  from  OPTEVFOR,  the  GTT 
requested  trained  Fleet  NMT  Operators/Maintainers  to  operate  the  NMTs  throughout  testing. 
This  same  information  was  in  the  6  April  2011  NMT  Development  Test/Integrated  Test  Cl  site 
briefs.  In  addition,  this  information  was  in  the  4  March  2010  (REV  A)  version  of  the  Test  and 
Evaluation  Master  Plan  (TEMP).  The  Navy  provided  the  briefings  and  the  approved  TEMP  to 
the  DoD  IG. 

In  addition,  the  first  sentence  of  the  first  paragraph  following  Table  1  is  incomplete.  See  the 
response  to  DoD  IG  Findings  under  “Results  In  Brief  and  Finding  A”  above. 
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2.  “SPAWAR  Canceled  Policy  on  Waivers  and  Deferrals”  (Draft  Report  Page  9) 

The  deficiencies  in  managing  waiver  requests  and  presenting  insufficient  OTRR  briefings 
occurred  because  Navy  policy  did  not  clearly  state  that  program  managers  must  request  waivers 
whenever  their  program  does  not  meet  all  criteria  required  to  start  IOT&E.  In  May  2009, 
SPAWAR  Notice  3960  canceled  the  SPAWAR  Instruction  3960.3E  which  included  a  checklist 
for  certification  of  readiness  for  operational  evaluation.  The  SPAWAR  Notice  3960  does  not 
include  any  guidance  on  the  OTRR  process  or  when  and  how  to  request  waivers  and  deferrals. 

The  PEO,  Command,  Control,  Communications,  Computers  and  Intelligence  and  Space,  had 
internal  guidance  that  had  an  OTRR  checklist  that  listed  14  of  the  20  IOT&E  certification  criteria 
identified  in  SECNAVINST  5000.2E.  However,  according  to  SPAWAR  staff,  the  PEO 
Operations  Guide  superseded  this  internal  guidance  in  2008.  Appendix  C  compares  the 
SECNAV  and  PEO  policies  and  identifies  the  six  criteria  the  PEO  policy  does  not  include. 

OPNAV  Response: 

N84:  Do  not  concur 

See  comments  above  under  “Finding  A” 

SPAWAR  Response: 

PEO  C4I:  Do  not  concur. 

The  finding  is  not  current  or  accurate  because  the  referenced  ’’internal  guidance”  was  a  draft 
policy  that  was  never  signed  nor  promulgated  to  the  program  offices.  Programs  did  not  have 
access  to  the  draft  policy.  Therefore,  the  comparisons  made  in  Appendix  C  are  not  relevant  as 
they  compare  an  unsigned,  unpromulgated  policy  memo  to  the  SECNAVINST  5000.2  criteria. 

In  addition,  the  comparison  to  the  draft  policy  memo  is  not  relevant  because  the  approved  PEO 
Operations  Guide  (POG)  was  promulgated  before  any  of  the  PEO  C4I  programs  were  ready  for 
IOT&E. 

3.  Conclusion  (Draft  Report  Page  10) 

The  NMT  entered  and  completed  IOT&E  with  unresolved  deficiencies  that  affected  software 
maturity  and  reliability.  NMT  did  not  fully  meet  its  primary  requirements  providing  Navy  forces 
worldwide  secure,  protected,  and  survivable  AEHF  satellite  communications.  At  the  final 
production  decision,  the  program  manager  accepted  77  NMTs  that  cost  $304.1  million  and 
whose  reliability  limited  their  ability  meeting  communication  needs  for  supporting  the  Navy’s 
long-term  deployments. 

Additionally,  because  the  program  managers  for  the  four  programs  did  not  prepare  documentation 
for  the  OTRR  briefings  that  captured  program  accomplishments  against  each  certification 
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requirement,  the  PEO  and  other  stakeholders  could  not  fully  consider  program  readiness  for 
entering  IOT&E  during  the  OTRR  briefing. 

OPNAV  Response: 

N84:  Do  not  concur 

As  stated  above,  the  OTRR  brief  is  the  final  action  of  an  extensive  and  robust  review  process. 
OTRR’s  are  focused  on  on  items  that  merit  PEO  attention  and  where  a  decision  is  required.  The 
prebriefing  process,  overseen  by  a  SES,  addresses  all  certification  issues  and  focuses  the  OTRR 
agenda. 


SPAWAR  Response: 

PEO  C4I/PMW  170/A:  Do  not  concur. 

The  finding  is  not  accurate  and  incomplete  because  the  NMT  TEMP  1642  REV  B,  signed  by  all 
stakeholders  and  approved  by  OSD  DOT&E  on  12  August  201 1  (para.  3.3. 6. 1.4,  pg.  3-16), 
documented  that  testing,  prior  to  an  available  AEHF  satellite,  could  evaluate  NMT’s 
compatibility,  functionality,  and  whether  it  could  perform  its  operational  missions.  For  example, 
NMT  successfully  tested  Low  Data  Rate  (LDR)  and  Medium  Data  Rate  (MDR)  capabilities  at 
IOT&E.  Based  on  the  COMOPTEVFOR  NMT  OT-C1  (IOT&E)  Test  Report  dated 
29  November  201 1  (Section  1.5.3,  Page  9): 

’’The  success  rate  of  LDR  and  MDR  Mission  Data  Update  (MDU)  Point-to-Point  (PTP) 
data  transfers  conducted  by  USS  COLE  with  USS  HUE  CITY  during  OT-C1  and  with 
Cruise  Missile  Support  Activity  during  IT-C1  was  100  percent.  MDU  data  transfers 
were  also  demonstrated  between  USS  ANNAPOLIS  and  the  Washington  Planning 
Center  using  ’’strike  over  secret”  with  a  100-percent  success  rate.  Sixty-eight  point  five 
percent  of  all  circuit  activations,  99.5  percent  of  all  circuit  deactivations,  80.0  percent 
of  the  network  joins,  and  98.6  percent  of  all  network  exits  were  successfully 
accomplished.  SRB  [Submarine  Report  Back]  processing  time  and  success  rate 
exceeded  the  classified  threshold  values,  and  the  results  are  summarized  in  table  B-7. 

The  NMT  successfully  demonstrated  the  capability  to  effectively  support  operational 
missions  on  all  available  beams  and  configurations  and  is  evaluated  as  satisfactory.” 

The  NMT  program  manager  received  direction  to  document  the  plan  to  test  the  Extended  Data 
Rate  (XDR)  capability  during  FOT&E  in  the  NMT  TEMP.  The  plan  to  test  is  in  both  REV  A 
(dated  23  November  2009;  approved  4  March  2010,  PRIOR  to  OTRR)  and  REV  B  of  the  NMT 
TEMP  approved  by  OSD  DOT&E  on  12  August  2011.  DOT&E  approval  of  the  IOT&E  plan 
was  with  full  knowledge  of  this  limitation.  The  Navy  provided  these  TEMPs  to  DoD  IG. 
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Also  see  the  Navy  response  to  DoD  IG  Findings  at  RESULTS  IN  BRIEF  above  regarding 
paragraph  2  of  the  Conclusion. 


4.  Recommendation  (Draft  Report  Page  10) 

We  recommend  the  Commander,  Space  and  Naval  Warfare  Systems  Command,  ensure  that  the 
Space  and  Naval  Warfare  Instruction  3960. 3F,  “Test  and  Evaluation,”  references  and  provides 
guidance  to  implement  the  Navy  policy  in  the  planned  revision  of  Secretary  of  the  Navy 
Instruction  5000.2E,  “Department  of  the  Navy  Implementation  and  Operation  of  the  Defense 
Acquisition  System  and  the  Joint  Capabilities  Integration  and  Development  System.” 

OPNAV  Response: 

N84:  Concur 

SPAWAR  Response:  Concur. 

SPAWAR  plans  to  promulgate  an  updated  Test  and  Evaluation  policy  (SPAWARINST  3960.5) 
that  will  addresses  roles  and  responsibilities  and  will  also  include  appropriate  direction  for 
SPAWAR  programs  to  follow  the  OTRR  waiver  process  as  documented  in  SECNAVINST 
5000.2E.  The  updated  SPAWARINST  3960.5  is  expected  to  be  released  by  November  2015. 


FINDING  B:  Navy  Systems  Commands  Need  to  Improve  Management  of  Waivers  and 

Deferrals  (Draft  Report  Page  11) 

Navy  program  managers  and  weapon  system  sponsors  within  the  Office  of  the  CNO  did  not  fully 
implement  Navy  policies  for  requesting  waivers  and  deferrals  before  certifying  program 
readiness  for  IOT&E  supporting  the  final  production  decision.  Our  review  of  nine  Navy 
acquisition  programs  that  entered  final  production  between  April  14,  2012,  and  April  14,  2014, 
showed  that  program  managers  on: 

•  four  programs  did  not  request  waivers  when  those  programs  did  not  meet  all  IOT&E 
certification  requirements; 

•  five  programs  had  OTRR  briefings  that  did  not  clearly  document  what  they  had  done  to  meet 
certification  criteria;  and 

•  one  program  did  not  request  deferrals  from  testing  planned  to  demonstrate  system 
requirements  during  IOT&E. 

These  conditions  occurred  because  Navy  policy  did  not  clearly  state  that  program  managers  must 
request  a  waiver  whenever  a  program  does  not  meet  all  criteria  required  to  enter  IOT&E  and 
request  a  deferral  to  delay  testing. 
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Additionally,  Navy  weapon  system  sponsors  for  three  programs  did  not  obtain  agreement  from 
the  Joint  Chiefs  of  Staff  (JCS)  that  the  waiver  or  deferral  would  not  affect  military  use  before 
independently  granting  the  waiver  or  deferral.  This  occurred  because: 

•  Navy  and  JCS  policy  did  not  clearly  define  responsibilities  for  designating  the  most  mission- 
critical  system  characteristics  as  primary  requirements;  and 

•  Navy  policy  did  not  require  sponsors  to  communicate  with  the  JCS  their  decision  to  grant  a 
waiver  or  deferral  to  a  program  designated  as  Acquisition  Category  I,  or  of  JCS  interest. 

As  a  result,  program  manager  and  sponsor  implementation  of  Navy  waiver  and  deferral  policy 
permitted  six  of  nine  programs  reviewed  to  complete  IOT&E  with  unresolved  deficiencies  that 
negatively  impacted  performing  the  primary  missions. 

OPNAV  Response: 

N84/N803:  Do  not  concur 

Finding  B  implies  that  the  only  way  a  PM/PEO  is  informed  of  a  system’s  required  capabilities 
and  determines  if  a  system  is  ready  to  proceed  to  operational  testing  is  through  reading  a  detailed 
requirement  document.  This  incorrectly  assumes  that  the  requirement  document  is  the  only 
vehicle  for  conveying  programmatic  information  to  the  PEO  and  PM.  Resource  sponsors,  PEOs 
and  PMs  are  in  daily  communications  discussing  risk,  cost,  performance  and  schedule. 

Numerous  technical  and  fiscal  realities,  as  well  as  actual  system  performance,  all  affect  the 
decision  making  process  associated  OTRRs. 

Existing  SECNAVINST  5000.2E  policy  paragraph  4. 6.2. 2  states  that  a  PEO  must  execute  the 
following  policy  when  certifying  a  system  to  proceed  to  OT  with  Exceptions  -  which 
encompasses  both  waivers  and  deferrals  (italics  added  for  accent): 

”4. 6.2. 2  Certification  for  OT  With  T&E  Exceptions 

Certify  to  CNO  ((N84),  DC,  CD&I  by  message  that  a  system  is  ready  for  OT(specific 
operational  test  phase),  as  required  by  the  TEMP,  with  waiver  and  or  deferral  requests.  Provide 
information  copies  to  the  program  sponsor  (Navy  only,  who  must  provide  formal  concurrence 
with  proposed  exceptions),  ASN(RD&A),  COMOPTEVFOR  and  Director,  MCOTEA,  and  when 
a  program  is  on  the  OSD  T&E  oversight  list,  to  DOT&E.” 

This  policy  ensures  that  the  PEO  delegated  Title  10  responsibility  from  the  CNO  to  develop 
requirements  for  systems  that  will  equip  the  fleet  have  executed  that  responsibility  in 
determining  that  the  system  capabilities  are  adequate  to  proceed  to  operational  testing.  The  Navy 
views  this  as  adequate  oversight. 
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Adding  a  requirement  for  a  waiver  or  deferral  to  be  approved  by  the  JROC  would  require 
significant  effort  (e.g.  multiple  flag  level  briefings)  before  the  waiver  could  even  start  the  Joint 
Staffing  process.  This  is  a  significant  burden  that  is  contradictory  to  CNO  direction  and  USD 
AT&L  BBP  3.0  guidance. 


SPAWAR  Response: 

PEO  C4I/PMW  170/A:  Do  not  concur. 

The  finding  is  not  accurate  and  incomplete.  See  the  SPAWAR  response  to  “FINDING  A”. 


APPENDIX  C:  Comparison  of  Certification  Criteria  in  SECNAV  and  SPAWAR  Policies 

(Draft  Report  Pages  25-28) 

SPAWAR  Response: 

PEO  C4I:  Do  Not  Concur. 

See  the  SPAWAR  response  above  to  “SPAWAR  Canceled  Policy  on  Waivers  and  Deferrals.” 


OVERALL  COMMENTS: 

1 .  The  three  day  staffing  time  for  the  Discussion  Draft  is  not  adequate  to  allow  full  review 
and  staffing,  especially  when  the  Discussion  Draft  is  delivered  without  any  warning.  Following 
the  Discussion  Draft,  the  final  IG  Draft  draft  was  provided  with  a  30  day  review  period. 

The  Navy  recommends  that  either:  1)  The  DODIG  eliminate  the  initial  discussion  draft 
review  and  only  conduct  a  formal  30  day  review,  with  a  formal  comment  adjudication  process  as 
discussed  below,  or  2)  The  DODIG  extend  the  timeline  for  the  initial  review  to  seven  days  to 
provide  the  Navy  adequate  time  to  thoroughly  review  and  comment  on  the  initial  decision  draft 
report. 

At  a  minimum,  the  audited  organizations  should  be  informed  of  the  intended  delivery  date 
and  schedule  so  that  they  can  be  prepared  to  staff  the  draft  report. 

2.  Similar  to  the  recent  NAVSEA  IG  report  on  Waivers  and  Deferrals,  the  Navy  is 
concerned  that  previously  provided  comments  (some  were  factual  in  nature)  to  Discussion  Draft 
reports  were  not  incorporated  into  the  final  Draft  report,  and  no  comment  adjudication 
opportunity  was  conducted  to  ensure  that  the  DODIG  understood  the  comments.  It  was 
determined  that  the  omissions  were  a  combination  of  administrative  errors,  DODIG  confusion 
regarding  interpretation  of  Navy  comments,  and  differences  of  opinion  between  DODIG  and 
Navy.  It  is  recommended  that  the  DODIG  implement  a  final  comment  adjudication  review  as 
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Department  of  the  Navy  (cont'd) 


NAVY  RESPONSE  TO 

DODIG  DRAFT  AUDIT  REPORT  “SPACE  AND  NAVAL  WARFARE  SYSTEMS 
COMMAND  AND  OVERALL  NAVY  NEEDS  TO  IMPROVE  MANAGEMENT 
OF  WAIVER  DEFERRAL  REQUESTS” 

D2015-D000AE-0 157.000,  DATED  2  JULY  2015 

standard  procedure  to  allow  the  audited  organization  to  understand  how  comments  were 
adjudicated  and  allow  for  better  understanding  on  why  comments  were  not  incorporated  into  the 
report. 

CLASSIFICATION  REVIEW 

N84  is  still  confirming  the  correct  classification  of  these  inputs. 
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Acronyms  and  Abbreviations 


Acronyms  and  Abbreviations 


AEHF 

Advanced  Extremely  High  Frequency 

CND 

Computer  Network  Defense 

CNO 

Chief  of  Naval  Operations 

DMR 

Digital  Modular  Radio 

DTS 

Distributed  Targeting  System 

IOT&E 

Initial  Operation  Test  and  Evaluation 

IDS 

Identity  Dominance  System 

JCIDS 

Joint  Capabilities  Integration  and  Development  System 

JCS 

Joint  Chiefs  of  Staff 

NAVAIR 

Naval  Air  Systems  Command 

NAVSEA 

Naval  Sea  Systems  Command 

NMT 

Navy  Multiband  Terminal 

OTRR 

Operational  Test  Readiness  Review 

PEO 

Program  Executive  Officer 

SECNAVINST 

Secretary  of  the  Navy  Instruction 

SM-6 

Standard  Missile-6 

SPAWAR 

Space  and  Naval  Warfare  Systems  Command 

TEMP 

Test  and  Evaluation  Master  Plan 
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Whistleblower  Protection 
U.S.  Department  of  Defense 


The  Whistleblower  Protection  Enhancement  Act  of  2012  requires 
the  Inspector  General  to  designate  a  Whistleblower  Protection 
Ombudsman  to  educate  agency  employees  about  prohibitions 
on  retaliation,  and  rights  and  remedies  against  retaliation  for 
protected  disclosures.  The  designated  ombudsman  is  the  DoD  Hotline 
Director.  For  more  information  on  your  rights  and  remedies  against 
retaliation,  visit  www.dodig.mil/programs/whistleblower. 


For  more  information  about  DoD  IG 
reports  or  activities,  please  contact  us: 

Congressional  Liaison 

congressional@dodig.mil;  703.604.8324 

Media  Contact 

public.affairs@dodig.mil;  703.604.8324 

For  Report  Notifications 

http://www.dodig.mil/pubs/email_update.cfm 

Twitter 

twitter.com/DoD_IG 


DoD  Hotline 

dodig.mil/hotline 
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